Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Tue, Sep 7, 2010 at 6:02 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2010-09-06 at 22:32 +0200, Boszormenyi Zoltan wrote: (in commit) write wal record release locks/etc   xact2 can proceed from here wait for sync ack In the first case, the contention is obviously increased.

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Boszormenyi Zoltan
Fujii Masao írta: On Tue, Sep 7, 2010 at 6:02 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2010-09-06 at 22:32 +0200, Boszormenyi Zoltan wrote: (in commit) write wal record release locks/etc xact2 can proceed from here wait for sync ack In the first case, the contention

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Wed, Sep 8, 2010 at 7:04 PM, Boszormenyi Zoltan z...@cybertec.at wrote: Then there is no use to implement individual sync/async replicated transactions, period. An async replicated transaction that waits for a sync replicated transaction because of locks will become implicitely sync. It

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Boszormenyi Zoltan
Fujii Masao írta: On Wed, Sep 8, 2010 at 7:04 PM, Boszormenyi Zoltan z...@cybertec.at wrote: Then there is no use to implement individual sync/async replicated transactions, period. An async replicated transaction that waits for a sync replicated transaction because of locks will become

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 6:52 AM, Boszormenyi Zoltan z...@cybertec.at wrote: Fujii Masao írta: On Wed, Sep 8, 2010 at 7:04 PM, Boszormenyi Zoltan z...@cybertec.at wrote: Then there is no use to implement individual sync/async replicated transactions, period. An async replicated transaction

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Wed, Sep 8, 2010 at 8:43 PM, Robert Haas robertmh...@gmail.com wrote: I still don't see why it matters whether you wait before or after releasing locks.  As soon as the transaction is marked committed in CLOG, other transactions can potentially see its effects. AFAIR, even if CLOG has been

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Markus Wanner
On 09/08/2010 12:04 PM, Boszormenyi Zoltan wrote: Then there is no use to implement individual sync/async replicated transactions, period. I disagree. Different transactions have different priorities for latency vs. failure-resistance. An async replicated transaction that waits for a sync

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Simon Riggs
On Wed, 2010-09-08 at 12:04 +0200, Boszormenyi Zoltan wrote: I know why you want that, but I don't think its right. Agreed. That's why I put the wait before ProcArrayEndTransaction() is called. Then there is no use to implement individual sync/async replicated

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 8:30 AM, Fujii Masao masao.fu...@gmail.com wrote:  And in any event, there is ALWAYS a window of time during which the client doesn't know the transaction has committed but other transactions can potentially see its effects. Yep. The problem here is that synchronous

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Fujii Masao
On Wed, Sep 8, 2010 at 10:07 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Sep 8, 2010 at 8:30 AM, Fujii Masao masao.fu...@gmail.com wrote:  And in any event, there is ALWAYS a window of time during which the client doesn't know the transaction has committed but other transactions can

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 9:32 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 8, 2010 at 10:07 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Sep 8, 2010 at 8:30 AM, Fujii Masao masao.fu...@gmail.com wrote:  And in any event, there is ALWAYS a window of time during which the client

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Simon Riggs
On Wed, 2010-09-08 at 09:50 -0400, Robert Haas wrote: So that means we have to make sure that none of the effects of a transaction can be seen until WAL is fsync'd on the master AND the slave has acked. Yes, that's right. And I like your example; one for the docs. There is a slight

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread David Fetter
On Wed, Sep 08, 2010 at 03:22:46PM +0100, Simon Riggs wrote: On Wed, 2010-09-08 at 09:50 -0400, Robert Haas wrote: So that means we have to make sure that none of the effects of a transaction can be seen until WAL is fsync'd on the master AND the slave has acked. Yes, that's right. And

Re: [HACKERS] Synchronization levels in SR

2010-09-08 Thread Robert Haas
On Wed, Sep 8, 2010 at 10:22 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-09-08 at 09:50 -0400, Robert Haas wrote: So that means we have to make sure that none of the effects of a transaction can be seen until WAL is fsync'd on the master AND the slave has acked. Yes, that's

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
Hi, On 05/27/2010 01:28 PM, Robert Haas wrote: How do you propose to guarantee that? ISTM that you have to either commit locally first, or send the commit to the remote first. Either way, the two events won't occur exactly simultaneously. I'm not getting the point of this discussion. As

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 4:01 AM, Markus Wanner mar...@bluegap.ch wrote: In any case, a server failure in between the commit request of the client and the commit confirmation for the client results in a client that can't tell if its transaction committed or not. So why do we care what to do

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 02:16 PM, Robert Haas wrote: Right, definitely. The trouble is that if they happen concurrently, and there's a crash, you have to be prepared for the possibility that either one of the two has completed and the other is not. Understood. In practice, this means that the master

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 9:45 AM, Markus Wanner mar...@bluegap.ch wrote: On 09/07/2010 02:16 PM, Robert Haas wrote: Right, definitely.  The trouble is that if they happen concurrently, and there's a crash, you have to be prepared for the possibility that either one of the two has completed and

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be taken. The slave isn't corrupt. It would

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Ron Mayer
Markus Wanner wrote: On 09/07/2010 02:16 PM, Robert Haas wrote: practice, this means that the master and standby need to compare notes on the ending WAL location and whichever one is further advanced needs to stream the intervening records to the other. Not necessarily, no. Remember that

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 16:31 +0200, Markus Wanner wrote: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Tom Lane
Markus Wanner mar...@bluegap.ch writes: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore the slave is now corrupt and a new base backup must be

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 11:17 -0400, Tom Lane wrote: Markus Wanner mar...@bluegap.ch writes: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead of the master location; therefore

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 04:47 PM, Ron Mayer wrote: In that situation, wouldn't it be possible that a different client queried the slave and already saw the result of that transaction which would later be rolled back? Good point, yes. Regards Markus Wanner -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 11:17 -0400, Tom Lane wrote: We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master. The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned.

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
Hi, On 09/07/2010 05:17 PM, Tom Lane wrote: Oh yes it is. If the slave replays WAL that didn't happen on the master, it might for instance have heap tuples in TID slots that are empty on the master, or index pages laid out differently from the master. Trying to apply additional WAL from the

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 11:06 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-09-07 at 16:31 +0200, Markus Wanner wrote: On 09/07/2010 04:15 PM, Robert Haas wrote: In theory, that's true, but if we do that, then there's an even bigger problem: the slave might have replayed WAL ahead

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
Hi, On 09/07/2010 06:00 PM, Robert Haas wrote: People who are more concerned about performance than robustness aren't going to use sync rep in the first place. I'm advocating sync (or eager, FWIW) replication for years, now. One of the hardest preconception I'm always confronted with is:

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Markus Wanner
On 09/07/2010 05:55 PM, Markus Wanner wrote: Robert's argument Sorry, I meant Ron. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread marcin mank
On Tue, Sep 7, 2010 at 5:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master.  The only way to make that safe is the compare-notes-and-ship-WAL-back approach that Robert mentioned. If you feel that

Re: [HACKERS] Synchronization levels in SR

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 4:06 PM, marcin mank marcin.m...@gmail.com wrote: On Tue, Sep 7, 2010 at 5:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: We can *not* allow the slave to replay WAL ahead of what is known committed to disk on the master.  The only way to make that safe is the

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Dimitri Fontaine
Boszormenyi Zoltan z...@cybertec.at writes: Sorry for answering such an old mail, but what is the purpose of a transaction level synchronous behaviour if async transactions can be held back by a sync transaction? I don't understand why it would be the case (sync holding back async

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Boszormenyi Zoltan
Dimitri Fontaine írta: Boszormenyi Zoltan z...@cybertec.at writes: Sorry for answering such an old mail, but what is the purpose of a transaction level synchronous behaviour if async transactions can be held back by a sync transaction? I don't understand why it would be the case

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: Dimitri Fontaine írta: Boszormenyi Zoltan z...@cybertec.at writes: Sorry for answering such an old mail, but what is the purpose of a transaction level synchronous behaviour if async transactions can be held back by a sync

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: The locks held by a transaction are released after RecordTransactionCommit(), and waiting for the sync ack happens in this function. Now what happens when a sync transaction hold a lock that

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Boszormenyi Zoltan
Simon Riggs írta: On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: Dimitri Fontaine írta: Boszormenyi Zoltan z...@cybertec.at writes: Sorry for answering such an old mail, but what is the purpose of a transaction level synchronous behaviour if async

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 22:32 +0200, Boszormenyi Zoltan wrote: (in commit) write wal record release locks/etc xact2 can proceed from here wait for sync ack In the first case, the contention is obviously increased. With this, we are creating more idle time in the server instead of letting

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Greg Stark
On Mon, Sep 6, 2010 at 10:02 PM, Simon Riggs si...@2ndquadrant.com wrote: Then I respectfully suggest that you're releasing locks too early. Your proposal would allow a 2nd user to see the results of the 1st user's transaction before the 1st user knew about whether it had committed or not.

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 23:07 +0100, Greg Stark wrote: On Mon, Sep 6, 2010 at 10:02 PM, Simon Riggs si...@2ndquadrant.com wrote: Then I respectfully suggest that you're releasing locks too early. Your proposal would allow a 2nd user to see the results of the 1st user's transaction before

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 16:11 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Mon, 2010-09-06 at 21:45 +0200, Boszormenyi Zoltan wrote: The locks held by a transaction are released after RecordTransactionCommit(), and waiting for the sync ack happens in this function.

Re: [HACKERS] Synchronization levels in SR

2010-09-06 Thread Robert Haas
On Mon, Sep 6, 2010 at 5:02 PM, Simon Riggs si...@2ndquadrant.com wrote: Then I respectfully suggest that you're releasing locks too early. Your proposal would allow a 2nd user to see the results of the 1st user's transaction before the 1st user knew about whether it had committed or not.

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Boszormenyi Zoltan
Hi, Dimitri Fontaine írta: Simon Riggs si...@2ndquadrant.com writes: Seems strange. If you have 2 standbys and you say you would like node1 to be the preferred candidate, then you load it so heavily that a remote server with by-definition much larger network delay responds first, then I

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Fujii Masao
On Fri, Sep 3, 2010 at 6:43 PM, Boszormenyi Zoltan z...@cybertec.at wrote: In my patch, when the transactions were waiting for ack from the standby, they have already released all their locks, the wait happened at the latest possible point in CommitTransaction(). In Fujii's patch (I am

Re: [HACKERS] Synchronization levels in SR

2010-09-03 Thread Boszormenyi Zoltan
Fujii Masao írta: On Fri, Sep 3, 2010 at 6:43 PM, Boszormenyi Zoltan z...@cybertec.at wrote: In my patch, when the transactions were waiting for ack from the standby, they have already released all their locks, the wait happened at the latest possible point in CommitTransaction(). In

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread David Fetter
On Thu, Jun 03, 2010 at 10:57:05PM -0400, Robert Haas wrote: On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck janwi...@yahoo.com wrote: What would be the use case for such a query? Monitoring? s/\?/!/; Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread Jan Wieck
On 6/3/2010 10:57 PM, Robert Haas wrote: On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck janwi...@yahoo.com wrote: On 5/27/2010 4:31 PM, Bruce Momjian wrote: Also, what would be cool would be if you could run a query on the master to view the SR commit mode of each slave. What would be the use

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread Robert Haas
On Fri, Jun 4, 2010 at 3:35 PM, Jan Wieck janwi...@yahoo.com wrote: On 6/3/2010 10:57 PM, Robert Haas wrote: On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck janwi...@yahoo.com wrote: On 5/27/2010 4:31 PM, Bruce Momjian wrote: Also, what would be cool would be if you could run a query on the

Re: [HACKERS] Synchronization levels in SR

2010-06-04 Thread Jan Wieck
On 6/4/2010 4:22 PM, Robert Haas wrote: On Fri, Jun 4, 2010 at 3:35 PM, Jan Wieck janwi...@yahoo.com wrote: So that justifies adding code, that the community needs to maintain and document, to the core system. If only I could find some monitoring case for transaction commit orders ... sigh!

Re: [HACKERS] Synchronization levels in SR

2010-06-03 Thread Jan Wieck
On 5/27/2010 4:31 PM, Bruce Momjian wrote: Heikki Linnakangas wrote: BTW, I think we're going to need a separate config file for listing the standbys anyway. There you can write per-server rules and options, but explicitly knowing about all the standbys also allows the master to recycle WAL

Re: [HACKERS] Synchronization levels in SR

2010-06-03 Thread Robert Haas
On Thu, Jun 3, 2010 at 8:47 PM, Jan Wieck janwi...@yahoo.com wrote: On 5/27/2010 4:31 PM, Bruce Momjian wrote: Heikki Linnakangas wrote: BTW, I think we're going to need a separate config file for listing the standbys anyway. There you can write per-server rules and options, but explicitly

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Greg Smith
Heikki Linnakangas wrote: The possibilities are endless... Your proposal above covers a pretty good set of scenarios, but it's by no means complete. If we try to solve everything the configuration will need to be written in a Turing-complete Replication Description Language. We'll have to pick

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Heikki Linnakangas
On 02/06/10 10:22, Greg Smith wrote: Heikki Linnakangas wrote: The possibilities are endless... Your proposal above covers a pretty good set of scenarios, but it's by no means complete. If we try to solve everything the configuration will need to be written in a Turing-complete Replication

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Simon Riggs
On Wed, 2010-06-02 at 03:22 -0400, Greg Smith wrote: Heikki Linnakangas wrote: The possibilities are endless... Your proposal above covers a pretty good set of scenarios, but it's by no means complete. If we try to solve everything the configuration will need to be written in a

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: It's pretty scary to call a user-defined function at that point in transaction. Not so much pretty scary as zero chance of being accepted. And I do mean zero. regards, tom lane -- Sent via pgsql-hackers

Re: [HACKERS] Synchronization levels in SR

2010-06-02 Thread Greg Smith
Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: It's pretty scary to call a user-defined function at that point in transaction. Not so much pretty scary as zero chance of being accepted. And I do mean zero. I swear, you guys are such buzzkills some

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 11:28 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 10:20 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: I guess that dropping the support of #3 doesn't reduce complexity since the code of #3 is almost the same

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 02:18 +0300, Heikki Linnakangas wrote: On 27/05/10 01:23, Simon Riggs wrote: On Thu, 2010-05-27 at 00:21 +0300, Heikki Linnakangas wrote: On 26/05/10 23:31, Dimitri Fontaine wrote: d. choice of commit or rollback at timeout Rollback is not an option. There is no

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Heikki Linnakangas
On 27/05/10 09:51, Simon Riggs wrote: On Thu, 2010-05-27 at 02:18 +0300, Heikki Linnakangas wrote: In practice, hard synchronous don't return ever until the commit hits the standby behavior is rarely what admins actually want, because it's disastrous from an availability point of view. More

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Wed, May 26, 2010 at 10:37 PM, Simon Riggs si...@2ndquadrant.com wrote: If the remote server responded first, then that proves it is a better candidate for failover than the one you think of as near. If the two standbys vary over time then you have network problems that will directly affect

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 3:21 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-05-27 at 11:28 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 10:20 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: I guess that dropping the support

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Dimitri Fontaine
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 26/05/10 23:31, Dimitri Fontaine wrote: So if you want simplicity to admin, effective data availability and precise control over the global setup, I say go for: a. transaction level control of the replication level b.

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 10:09 +0300, Heikki Linnakangas wrote: No, not necessarily. As I said above, you might just want a guarantee that *if* you query the standby, you get up-to-date results. Of course. COMMIT was already one of the options, so this comment was already understood. What we are

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 16:35 +0900, Fujii Masao wrote: On Thu, May 27, 2010 at 3:21 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-05-27 at 11:28 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 10:20 PM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-05-26 at 18:52

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 16:13 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 10:37 PM, Simon Riggs si...@2ndquadrant.com wrote: Please explain what will happen when the near server is unavailable, with per-standby settings. Please also explain what will happen if we choose to have 4 or 5

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 16:13 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 10:37 PM, Simon Riggs si...@2ndquadrant.com wrote: If the remote server responded first, then that proves it is a better candidate for failover than the one you think of as near. If the two standbys vary over time

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 6:30 PM, Simon Riggs si...@2ndquadrant.com wrote: Who sends the ack message? walreceiver Who receives it? walsender Would it be easier to have this happen in a second pair of processes WALSynchroniser (on primary) and WAL Acknowledger (on standby). WALAcknowledger

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 19:21 +0900, Fujii Masao wrote: On Thu, May 27, 2010 at 6:30 PM, Simon Riggs si...@2ndquadrant.com wrote: Would it be easier to have this happen in a second pair of processes WALSynchroniser (on primary) and WAL Acknowledger (on standby). WALAcknowledger would send

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 7:21 PM, Simon Riggs si...@2ndquadrant.com wrote: Seems strange. If you have 2 standbys and you say you would like node1 to be the preferred candidate, then you load it so heavily that a remote server with by-definition much larger network delay responds first, then I

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 7:33 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-05-27 at 19:21 +0900, Fujii Masao wrote: On Thu, May 27, 2010 at 6:30 PM, Simon Riggs si...@2ndquadrant.com wrote: Would it be easier to have this happen in a second pair of processes WALSynchroniser (on

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Dimitri Fontaine
Simon Riggs si...@2ndquadrant.com writes: Seems strange. If you have 2 standbys and you say you would like node1 to be the preferred candidate, then you load it so heavily that a remote server with by-definition much larger network delay responds first, then I say your preference was wrong.

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On Thu, May 27, 2010 at 3:13 AM, Fujii Masao masao.fu...@gmail.com wrote: (1) most standard case: 1 master + 1 sync standby (near)    When the master goes down, something like a clusterware detects that    failure, and brings the standby online. Since we can ensure that the    standby has all

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 20:13 +0900, Fujii Masao wrote: On Thu, May 27, 2010 at 7:33 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-05-27 at 19:21 +0900, Fujii Masao wrote: On Thu, May 27, 2010 at 6:30 PM, Simon Riggs si...@2ndquadrant.com wrote: Would it be easier to have

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Simon Riggs
On Thu, 2010-05-27 at 19:50 +0900, Fujii Masao wrote: For now, I agree that we support a quorum commit feature for 9.1 or later. But I don't think that it's simpler, more intuitive and easier-to-understand than per-standby setting. So I think that we should include the per-standby setting in

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 8:28 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 27, 2010 at 3:13 AM, Fujii Masao masao.fu...@gmail.com wrote: (1) most standard case: 1 master + 1 sync standby (near)    When the master goes down, something like a clusterware detects that    failure, and

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 8:30 PM, Simon Riggs si...@2ndquadrant.com wrote: Why does it? I just explained a design where that wasn't required. Hmm.. my expression might have been ambiguous. Walreceiver still needs to wait for WAL flush by walwriter *before* sending the ACK to the master, in #3

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 8:30 PM, Simon Riggs si...@2ndquadrant.com wrote: There already is a first patch to the community that implements quorum commit, just not by you. Yeah, AFAIK, that patch includes also per-standby setting. If you have a better way, describe it in detail and in full now,

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On Thu, May 27, 2010 at 8:02 AM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, May 27, 2010 at 8:28 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 27, 2010 at 3:13 AM, Fujii Masao masao.fu...@gmail.com wrote: (1) most standard case: 1 master + 1 sync standby (near)    When the

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Fujii Masao
On Thu, May 27, 2010 at 9:48 PM, Robert Haas robertmh...@gmail.com wrote: There could still be additional transactions that the original master has committed locally but were not acked to the client.  I guess you'd just work around that by taking a new base backup from the new master. Right.

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On Thu, May 27, 2010 at 9:09 AM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, May 27, 2010 at 9:48 PM, Robert Haas robertmh...@gmail.com wrote: There could still be additional transactions that the original master has committed locally but were not acked to the client.  I guess you'd just

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Bruce Momjian
Simon Riggs wrote: On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: I guess that dropping the support of #3 doesn't reduce complexity since the code of #3 is almost the same as that of #2. Like walreceiver sends the ACK after receiving the WAL in #2 case, it has only to do the same

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Bruce Momjian
Heikki Linnakangas wrote: BTW, I think we're going to need a separate config file for listing the standbys anyway. There you can write per-server rules and options, but explicitly knowing about all the standbys also allows the master to recycle WAL as soon as it has been streamed to all the

Re: [HACKERS] Synchronization levels in SR

2010-05-27 Thread Robert Haas
On May 27, 2010, at 4:31 PM, Bruce Momjian br...@momjian.us wrote: Heikki Linnakangas wrote: BTW, I think we're going to need a separate config file for listing the standbys anyway. There you can write per-server rules and options, but explicitly knowing about all the standbys also allows

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Tue, 2010-05-25 at 23:59 -0400, Robert Haas wrote: Quorum commit is definitely an extra knob, IMHO. No, its about three less, as I have explained. Explain your position, don't just demand others listen. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 13:03 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 1:04 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-25 at 12:40 +0900, Fujii Masao wrote: On Tue, May 25, 2010 at 10:29 AM, Josh Berkus j...@agliodbs.com wrote: I agree that #4 should be done last,

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 12:36 +0900, Fujii Masao wrote: On Wed, May 26, 2010 at 2:10 AM, Simon Riggs si...@2ndquadrant.com wrote: My suggestion is simply to have a single parameter (name unimportant) number_of_synch_servers_we_wait_for = N which is much easier to understand because it is

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Fujii Masao
On Wed, May 26, 2010 at 5:02 PM, Simon Riggs si...@2ndquadrant.com wrote: Everything I've said about per-standby settings applies here, which was based upon having just 2 settings: sync and async. If you have four settings instead, things get even more complex. If we were going to reduce

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Robert Haas
On Wed, May 26, 2010 at 2:31 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-25 at 23:59 -0400, Robert Haas wrote: Quorum commit is definitely an extra knob, IMHO. No, its about three less, as I have explained. Explain your position, don't just demand others listen. OK. In

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Alastair Turner
A suggestion, based on what I believe would be ideal default settings for a fully developed SR capability. The thought being that as long as the default behaviour was stable additional knobs could be added across version boundaries without causing trouble. Per slave the master needs to know: -

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: I guess that dropping the support of #3 doesn't reduce complexity since the code of #3 is almost the same as that of #2. Like walreceiver sends the ACK after receiving the WAL in #2 case, it has only to do the same thing after the WAL

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 18:52 +0900, Fujii Masao wrote: To summarise, I think we can get away with just 3 parameters: synchronous_replication = N # similar in name to synchronous_commit synch_rep_timeout = T synch_rep_timeout_action = commit | abort I agree to add the latter two

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 07:10 -0400, Robert Haas wrote: OK. In words of one syllable, your way still has all the same knobs, plus some more. I explained how the per-standby settings would take many parameters, whereas per-transaction settings take far fewer. You sketched out a design which

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Robert Haas
On Wed, May 26, 2010 at 9:54 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, 2010-05-26 at 07:10 -0400, Robert Haas wrote: OK.  In words of one syllable, your way still has all the same knobs, plus some more. I explained how the per-standby settings would take many parameters, whereas

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 26/05/10 18:31, Robert Haas wrote: And frankly, I don't think it's possible for quorum commit to reduce the number of parameters. Even if we have that feature available, not everyone will want to use it. And the people who don't will presumably need whatever parameters they would have

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: One way to do that would be to refrain from flushing the commit record to disk on the master until the standby has acknowledged it. I'm not clear on the benefit of doing that, versus flushing the commit record and then waiting

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Heikki Linnakangas
On 26/05/10 20:10, Kevin Grittner wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com wrote: One way to do that would be to refrain from flushing the commit record to disk on the master until the standby has acknowledged it. I'm not clear on the benefit of doing that, versus

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 11:31 -0400, Robert Haas wrote: Your reply has again avoided the subject of how we would handle failure modes with per-standby settings. That is important. I don't think anyone is avoiding that, we just haven't discussed it. You haven't discussed it, but even before

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 12:10 -0500, Kevin Grittner wrote: Adding a synchronous standby should require some action in the master, since it affects the behavior on master. +1 +1 -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Although, if the master crashes at that point, and quickly recovers, you could see the last transactions committed on the master before they're replicated to the standby. Versus having the transaction committed on one or more

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Simon Riggs
On Wed, 2010-05-26 at 19:55 +0300, Heikki Linnakangas wrote: Now you want to set up a temporary replica of the master at a development server, for testing purposes. If you set quorum to 2, your development server becomes critical infrastructure, which is not what you want. That's a good

Re: [HACKERS] Synchronization levels in SR

2010-05-26 Thread Robert Haas
On Wed, May 26, 2010 at 1:24 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 26/05/10 20:10, Kevin Grittner wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com  wrote: One way to do that would be to refrain from flushing the commit record to disk on the master

  1   2   >