Re: [HACKERS] Synchronous replication

2010-12-13 Thread Robert Haas
On Wed, Sep 15, 2010 at 8:39 AM, Fujii Masao masao.fu...@gmail.com wrote: I rebased the patch against current HEAD because it conflicted with recent commits about a latch. Can you please rebase this again? It no longer applies. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-15 Thread Erik Rijkers
On Wed, September 15, 2010 11:58, Fujii Masao wrote: On Wed, Sep 15, 2010 at 6:38 AM, David Fetter da...@fetter.org wrote: Now that the latch patch is in, when do you think you'll be able to use it instead of the poll loop? Here is the updated version, which uses a latch in communication from

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-15 Thread Erik Rijkers
nevermind... I see standbys.conf is now used. sorry for the noise... Erik Rijkers On Thu, September 16, 2010 01:12, Erik Rijkers wrote: On Wed, September 15, 2010 11:58, Fujii Masao wrote: On Wed, Sep 15, 2010 at 6:38 AM, David Fetter da...@fetter.org wrote: Now that the latch patch is in,

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-14 Thread David Fetter
On Fri, Sep 10, 2010 at 11:52:20AM +0900, Fujii Masao wrote: On Fri, Sep 3, 2010 at 3:42 PM, Fujii Masao masao.fu...@gmail.com wrote: Here is the proposed detailed design: standbys.conf = # This is not initialized by initdb, so users need to create it under $PGDATA.    

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-09 Thread Fujii Masao
On Fri, Sep 10, 2010 at 11:52 AM, Fujii Masao masao.fu...@gmail.com wrote: The attached patch implements the above and simple synchronous replication feature, which doesn't include quorum commit capability. The replication mode (async, recv, fsync, replay) can be specified on a per-standby

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Heikki Linnakangas
On 06/09/10 17:14, Simon Riggs wrote: On Mon, 2010-09-06 at 16:14 +0300, Heikki Linnakangas wrote: The standby is sending a stream of messages to the master with current LSN positions at the time the message is sent. Given a synchronous transaction, the master would wait until the feedback

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 09:27 +0300, Heikki Linnakangas wrote: On 06/09/10 17:14, Simon Riggs wrote: On Mon, 2010-09-06 at 16:14 +0300, Heikki Linnakangas wrote: The standby is sending a stream of messages to the master with current LSN positions at the time the message is sent. Given a

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Heikki Linnakangas
On 07/09/10 12:47, Simon Riggs wrote: The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for apply while nothing else is received, we do. Ok, thank you. The obvious performance problem is that

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 13:11 +0300, Heikki Linnakangas wrote: The obvious performance problem Is not obvious at all, and you misunderstand again. This emphasises the need for me to show code. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Training and

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 09:27 +0300, Heikki Linnakangas wrote: For the sake of argument, yes that's what I was thinking. Now please explain how *you're* thinking it should work. The WAL is sent from master to standby in 8192 byte chunks, frequently

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 10:47 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 09:27 +0300, Heikki Linnakangas wrote: For the sake of argument, yes that's what I was thinking. Now please explain how *you're* thinking it should work. The WAL is sent

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 10:47 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From standby, one reply per chunk. If we need to wait for

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: Oh, well you certainly didn't explain it well then. What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 11:41 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Tue, 2010-09-07 at 10:47 -0400, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The WAL is sent from master to standby in 8192 byte chunks, frequently including multiple commits. From

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 11:59 AM, Simon Riggs si...@2ndquadrant.com wrote: What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream immediately before going to

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Simon Riggs
On Tue, 2010-09-07 at 12:07 -0400, Robert Haas wrote: On Tue, Sep 7, 2010 at 11:59 AM, Simon Riggs si...@2ndquadrant.com wrote: What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Robert Haas
On Tue, Sep 7, 2010 at 2:15 PM, Simon Riggs si...@2ndquadrant.com wrote: Every time I explain anything, I get someone run around shouting but that can't work!. I'm sorry, but again your logic is poor and the bias against properly considering viable alternatives is the only thing perfectly

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-07 Thread Bruce Momjian
Robert Haas wrote: On Tue, Sep 7, 2010 at 11:59 AM, Simon Riggs si...@2ndquadrant.com wrote: What I *think* you're saying is that the slave doesn't send per-commit messages, but instead processes the WAL as it's received and then sends a heres-where-I-am status message back upstream

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-06 Thread Dimitri Fontaine
Disclaimer : I have understood things in a way that allows me to answer here, I don't know at all if that's the way it's meant to be understood. Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: (scratches head..) What's the point of differentiating received/fsynced/replayed, if

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-06 Thread Heikki Linnakangas
On 06/09/10 16:03, Dimitri Fontaine wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: (scratches head..) What's the point of differentiating received/fsynced/replayed, if the master receives the ack for all of them at the same time? It wouldn't the way I understand Simon's

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-06 Thread Simon Riggs
On Mon, 2010-09-06 at 16:14 +0300, Heikki Linnakangas wrote: The standby is sending a stream of messages to the master with current LSN positions at the time the message is sent. Given a synchronous transaction, the master would wait until the feedback stream reports that the current

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-06 Thread Robert Haas
On Mon, Sep 6, 2010 at 10:14 AM, Simon Riggs si...@2ndquadrant.com wrote: That doesn't really answer the question: *when* does standby send back the acknowledgment? I think you should explain when you think this happens in your proposal. Are you saying that you think the standby should send

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Simon Riggs
On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: On Thu, Sep 2, 2010 at 11:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I understand what you're after, the idea of being able to set synchronization level on a per-transaction basis is cool. But I haven't seen a

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Fujii Masao
On Thu, Sep 2, 2010 at 7:24 PM, Fujii Masao masao.fu...@gmail.com wrote: I propose a configuration file standbys.conf, in the master: # STANDBY NAME    SYNCHRONOUS   TIMEOUT importantreplica  yes           100ms tempcopy          no            10s Seems good. In fact, instead of yes/no,

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the standby should reply. If we choose async/recv/fsync/replay on a per-transaction basis, the standby should send multiple LSNs and the master needs to decide when

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Fujii Masao
On Fri, Sep 3, 2010 at 3:36 PM, Simon Riggs si...@2ndquadrant.com wrote: The former seems to be more useful, but triples the number of ACK from the standby. I'm not sure whether its overhead is ignorable, especially when the distance between the master and the standby is very long. No, it

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Simon Riggs
On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the standby should reply. If we choose async/recv/fsync/replay on a per-transaction basis, the standby

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 10:45, Simon Riggs wrote: On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the standby should reply. If we choose async/recv/fsync/replay on a

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Simon Riggs
On Fri, 2010-09-03 at 12:33 +0300, Heikki Linnakangas wrote: On 03/09/10 10:45, Simon Riggs wrote: On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote: That design would affect what the

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-03 Thread Heikki Linnakangas
On 03/09/10 13:20, Simon Riggs wrote: On Fri, 2010-09-03 at 12:33 +0300, Heikki Linnakangas wrote: On 03/09/10 10:45, Simon Riggs wrote: On Fri, 2010-09-03 at 09:55 +0300, Heikki Linnakangas wrote: On 03/09/10 09:36, Simon Riggs wrote: On Fri, 2010-09-03 at 12:50 +0900, Fujii Masao wrote:

Re: register/unregister standby Re: [HACKERS] Synchronous replication

2010-09-02 Thread Dimitri Fontaine
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Hmm, that's clever. I was thinking that you'd initialize the standby from an existing backup, and in that context the standby would not need to connect to the master except via the replication connection. To take a base backup,

Re: register/unregister standby Re: [HACKERS] Synchronous replication

2010-09-02 Thread Itagaki Takahiro
On Thu, Sep 2, 2010 at 6:41 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: In fact you don't need shell access here, it's rather easy to stream the base backup from the libpq connection, as implemented here :  http://github.com/dimitri/pg_basebackup There's been some talk of being able

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Fujii Masao
On Wed, Sep 1, 2010 at 7:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That requirement falls out from the handling of disconnected standbys. If a standby is not connected, what does the master do with commits? If the answer is anything else than acknowledge them to the

Re: register/unregister standby Re: [HACKERS] Synchronous replication

2010-09-02 Thread Thom Brown
On 30 August 2010 13:14, Fujii Masao masao.fu...@gmail.com wrote: I think that the advantage of registering standbys is that we can specify which WAL files the master has to keep for the upcoming standby. IMO, it's usually called together with pg_start_backup as follows:    SELECT

Re: register/unregister standby Re: [HACKERS] Synchronous replication

2010-09-02 Thread Dimitri Fontaine
Itagaki Takahiro itagaki.takah...@gmail.com writes:  http://github.com/dimitri/pg_basebackup There's been some talk of being able to stream a base backup over the replication connection too, which would be extremely handy. Yes please ! :) One issue of the base backup function is that the

Re: register/unregister standby Re: [HACKERS] Synchronous replication

2010-09-02 Thread Itagaki Takahiro
On Thu, Sep 2, 2010 at 7:54 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: One issue of the base backup function is that the operation will be a long transaction. So, non-transactional special commands, as like as VACUUM, would be better in terms of performance. For example, CREATE or

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Simon Riggs
On Thu, 2010-09-02 at 19:24 +0900, Fujii Masao wrote: On Wed, Sep 1, 2010 at 7:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That requirement falls out from the handling of disconnected standbys. If a standby is not connected, what does the master do with commits? If

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Heikki Linnakangas
On 02/09/10 15:03, Simon Riggs wrote: On Thu, 2010-09-02 at 19:24 +0900, Fujii Masao wrote: On Wed, Sep 1, 2010 at 7:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That requirement falls out from the handling of disconnected standbys. If a standby is not connected, what

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Simon Riggs
On Thu, 2010-09-02 at 15:15 +0300, Heikki Linnakangas wrote: On 02/09/10 15:03, Simon Riggs wrote: On Thu, 2010-09-02 at 19:24 +0900, Fujii Masao wrote: On Wed, Sep 1, 2010 at 7:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: That requirement falls out from the

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Robert Haas
On Thu, Sep 2, 2010 at 8:44 AM, Simon Riggs si...@2ndquadrant.com wrote: All standbys has no meaning without registration. It is not a question that needs an answer. Tell that to the DBA. I bet s/he knows what all standbys means. The fact that the system doesn't know something doesn't make it

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: Tell that to the DBA. I bet s/he knows what all standbys means. The fact that the system doesn't know something doesn't make it unimportant. Well as a DBA I think I'd much prefer to attribute votes to each standby so that each ack is weighted. Let me

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Simon Riggs
On Thu, 2010-09-02 at 08:59 -0400, Robert Haas wrote: On Thu, Sep 2, 2010 at 8:44 AM, Simon Riggs si...@2ndquadrant.com wrote: All standbys has no meaning without registration. It is not a question that needs an answer. Tell that to the DBA. I bet s/he knows what all standbys means. The

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Robert Haas
On Thu, Sep 2, 2010 at 10:06 AM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-09-02 at 08:59 -0400, Robert Haas wrote: On Thu, Sep 2, 2010 at 8:44 AM, Simon Riggs si...@2ndquadrant.com wrote: All standbys has no meaning without registration. It is not a question that needs an

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Heikki Linnakangas
On 02/09/10 17:06, Simon Riggs wrote: On Thu, 2010-09-02 at 08:59 -0400, Robert Haas wrote: On Thu, Sep 2, 2010 at 8:44 AM, Simon Riggssi...@2ndquadrant.com wrote: All standbys has no meaning without registration. It is not a question that needs an answer. Tell that to the DBA. I bet s/he

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Joshua Tolley
On Wed, Sep 01, 2010 at 04:53:38PM +0900, Fujii Masao wrote: - down When that situation occurs, the master shuts down immediately. Though this is unsafe for the system requiring high availability, as far as I recall, some people wanted this mode in the previous discussion. Oracle

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-02 Thread Fujii Masao
On Thu, Sep 2, 2010 at 11:32 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I understand what you're after, the idea of being able to set synchronization level on a per-transaction basis is cool. But I haven't seen a satisfactory design for it. I don't understand how it would

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-01 Thread Fujii Masao
On Wed, Sep 1, 2010 at 2:33 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Once we're done with that, all the big questions are still left. Yeah, let's discuss about those topics :) How to configure it? Before discussing about that, we should determine whether registering

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-01 Thread Heikki Linnakangas
On 01/09/10 10:53, Fujii Masao wrote: Before discussing about that, we should determine whether registering standbys in master is really required. It affects configuration a lot. Heikki thinks that it's required, but I'm still unclear about why and how. Why do standbys need to be registered in

Re: register/unregister standby Re: [HACKERS] Synchronous replication

2010-09-01 Thread Heikki Linnakangas
On 30/08/10 15:14, Fujii Masao wrote: I think that the advantage of registering standbys is that we can specify which WAL files the master has to keep for the upcoming standby. IMO, it's usually called together with pg_start_backup as follows: SELECT register_standby('foo',

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-01 Thread Robert Haas
On Wed, Sep 1, 2010 at 6:23 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I'm not sure what we should aim for in the first phase. But if you want as little code as possible yet have something useful, I think 'replay' mode with no standby registration is the way to go. IMHO,

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-01 Thread Simon Riggs
On Wed, 2010-09-01 at 08:33 +0300, Heikki Linnakangas wrote: On 01/09/10 04:02, Robert Haas wrote: See the thread on interruptible sleeps. The problem right now is that there are some polling loops that act to throttle the maximum rate at which a node doing sync rep can make forward

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-09-01 Thread Simon Riggs
On Wed, 2010-09-01 at 13:23 +0300, Heikki Linnakangas wrote: On 01/09/10 10:53, Fujii Masao wrote: Before discussing about that, we should determine whether registering standbys in master is really required. It affects configuration a lot. Heikki thinks that it's required, but I'm still

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread Bruce Momjian
fazool mein wrote: Hello everyone, I'm interested in benchmarking synchronous replication, to see how performance degrades compared to asynchronous streaming replication. I browsed through the archive of emails, but things still seem unclear. Do we have a final agreed upon patch that I

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread David Fetter
On Tue, Aug 31, 2010 at 05:44:15PM -0400, Bruce Momjian wrote: fazool mein wrote: Hello everyone, I'm interested in benchmarking synchronous replication, to see how performance degrades compared to asynchronous streaming replication. I browsed through the archive of emails, but

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread Robert Haas
On Tue, Aug 31, 2010 at 6:24 PM, David Fetter da...@fetter.org wrote: On Tue, Aug 31, 2010 at 05:44:15PM -0400, Bruce Momjian wrote: fazool mein wrote: Hello everyone, I'm interested in benchmarking synchronous replication, to see how performance degrades compared to asynchronous

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread Robert Haas
On Tue, Aug 31, 2010 at 6:24 PM, David Fetter da...@fetter.org wrote: On Tue, Aug 31, 2010 at 05:44:15PM -0400, Bruce Momjian wrote: fazool mein wrote: Hello everyone, I'm interested in benchmarking synchronous replication, to see how performance degrades compared to asynchronous

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread David Fetter
On Tue, Aug 31, 2010 at 08:34:31PM -0400, Robert Haas wrote: On Tue, Aug 31, 2010 at 6:24 PM, David Fetter da...@fetter.org wrote: On Tue, Aug 31, 2010 at 05:44:15PM -0400, Bruce Momjian wrote: fazool mein wrote: Hello everyone, I'm interested in benchmarking synchronous replication,

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread Robert Haas
On Tue, Aug 31, 2010 at 8:45 PM, David Fetter da...@fetter.org wrote: I am pretty sure, however, that the performance will be terrible at this point.  Heikki is working on fixing that, but it ain't done yet. Is this something for an eDB feature, or for community PostgreSQL, or...? It's an

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread Fujii Masao
On Wed, Sep 1, 2010 at 9:34 AM, Robert Haas robertmh...@gmail.com wrote: There are patches, and the latest from Fujii Masao is probably worth looking at :) I am pretty sure, however, that the performance will be terrible at this point.  Heikki is working on fixing that, but it ain't done yet.

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread fazool mein
Thanks! I'll wait for the merging then; there is no point in benchmarking otherwise. Regards On Tue, Aug 31, 2010 at 6:06 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 1, 2010 at 9:34 AM, Robert Haas robertmh...@gmail.com wrote: There are patches, and the latest from Fujii Masao

Re: [HACKERS] Synchronous replication - patch status inquiry

2010-08-31 Thread Heikki Linnakangas
On 01/09/10 04:02, Robert Haas wrote: See the thread on interruptible sleeps. The problem right now is that there are some polling loops that act to throttle the maximum rate at which a node doing sync rep can make forward progress, independent of the capabilities of the hardware. To be

register/unregister standby Re: [HACKERS] Synchronous replication

2010-08-30 Thread Fujii Masao
On Tue, Aug 10, 2010 at 5:58 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 05/08/10 17:14, Fujii Masao wrote: I'm thinking to make users register and unregister each standbys via SQL functions like register_standby() and unregister_standby(): The register/unregister

Re: [HACKERS] Synchronous replication

2010-08-16 Thread Heikki Linnakangas
On 05/08/10 13:40, Fujii Masao wrote: On Wed, Aug 4, 2010 at 12:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: There's some race conditions with the signaling. If another process finishes XLOG flush and sends the signal when a walsender has just finished one iteration of

Re: [HACKERS] Synchronous replication

2010-08-09 Thread Heikki Linnakangas
On 05/08/10 17:14, Fujii Masao wrote: I'm thinking to make users register and unregister each standbys via SQL functions like register_standby() and unregister_standby(): The register/unregister facility should be accessible from the streaming replication connection, so that you don't need to

Re: [HACKERS] Synchronous replication

2010-08-09 Thread Heikki Linnakangas
On 01/08/10 15:30, Greg Stark wrote: On Sun, Aug 1, 2010 at 7:11 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't think any of this quorum stuff makes much sense without explicitly registering standbys in the master. This doesn't have to be done manually. The

Re: [HACKERS] Synchronous replication

2010-08-09 Thread Heikki Linnakangas
I wonder if we can continue to rely on the pg_sleep() loop for sleeping in walsender. On those platforms where interrupts don't interrupt sleep, sending the signal is not going to promptly wake up walsender. That was fine before, but any delay is going to be poison to synchronous replication

Re: [HACKERS] Synchronous replication

2010-08-09 Thread Bruce Momjian
Fujii Masao wrote: On Wed, Aug 4, 2010 at 10:38 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Then you risk running out of disk space. Similar to having an archive command that fails for some reason. That's one reason the registration should not be too automatic -

Re: [HACKERS] Synchronous replication

2010-08-05 Thread Fujii Masao
On Wed, Aug 4, 2010 at 12:35 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: There's some race conditions with the signaling. If another process finishes XLOG flush and sends the signal when a walsender has just finished one iteration of its main loop, walsender will reset

Re: [HACKERS] Synchronous replication

2010-08-05 Thread Fujii Masao
On Wed, Aug 4, 2010 at 10:38 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Then you risk running out of disk space. Similar to having an archive command that fails for some reason. That's one reason the registration should not be too automatic - there is serious

Re: [HACKERS] Synchronous replication

2010-08-04 Thread Heikki Linnakangas
On 27/07/10 13:29, Fujii Masao wrote: On Tue, Jul 27, 2010 at 7:39 PM, Yeb Havingayebhavi...@gmail.com wrote: Fujii Masao wrote: I noted the changes in XlogSend where instead of *caughtup = true/false it now returns !MyWalSnd-sndrqst. That value is initialized to false in that procedure and it

Re: [HACKERS] Synchronous replication

2010-08-04 Thread Heikki Linnakangas
On 02/08/10 11:45, Fujii Masao wrote: On Sun, Aug 1, 2010 at 3:11 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't think any of this quorum stuff makes much sense without explicitly registering standbys in the master. I'm not sure if this is a good idea. This

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Fujii Masao
On Sun, Aug 1, 2010 at 3:11 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't think any of this quorum stuff makes much sense without explicitly registering standbys in the master. I'm not sure if this is a good idea. This requires users to do more manual operations than

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Fujii Masao
On Sun, Aug 1, 2010 at 9:51 PM, Robert Haas robertmh...@gmail.com wrote: Perhaps someone will claim that nobody wants to do that anyway (which I don't believe, BTW), but even in simpler cases it would be nicer to have an explicit policy rather than - in effect - inferring a policy from a soup

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Robert Haas
On Mon, Aug 2, 2010 at 5:02 AM, Fujii Masao masao.fu...@gmail.com wrote: On Sun, Aug 1, 2010 at 9:51 PM, Robert Haas robertmh...@gmail.com wrote: Perhaps someone will claim that nobody wants to do that anyway (which I don't believe, BTW), but even in simpler cases it would be nicer to have an

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Fujii Masao
On Mon, Aug 2, 2010 at 7:53 PM, Robert Haas robertmh...@gmail.com wrote: Let's not get *the manner of specifying the policy* confused with *the need to update the policy when the master changes*.  It doesn't seem likely you would want the same value for  synchronous_standbys on all your

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Robert Haas
On Mon, Aug 2, 2010 at 7:06 AM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Aug 2, 2010 at 7:53 PM, Robert Haas robertmh...@gmail.com wrote: Let's not get *the manner of specifying the policy* confused with *the need to update the policy when the master changes*.  It doesn't seem likely

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Fujii Masao
On Mon, Aug 2, 2010 at 8:32 PM, Robert Haas robertmh...@gmail.com wrote: Sure.  If you give the standbys names, then if people change the names, they'll have to update their configuration.  But I can't see that as an argument against doing it.  You can remove the possibility that someone will

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Yeb Havinga
Fujii Masao wrote: On Mon, Aug 2, 2010 at 7:53 PM, Robert Haas robertmh...@gmail.com wrote: Let's not get *the manner of specifying the policy* confused with *the need to update the policy when the master changes*. It doesn't seem likely you would want the same value for

Re: [HACKERS] Synchronous replication

2010-08-02 Thread Robert Haas
On Mon, Aug 2, 2010 at 8:57 AM, Yeb Havinga yebhavi...@gmail.com wrote: Fujii Masao wrote: On Mon, Aug 2, 2010 at 7:53 PM, Robert Haas robertmh...@gmail.com wrote: Let's not get *the manner of specifying the policy* confused with *the need to update the policy when the master changes*.  It

Re: [HACKERS] Synchronous replication

2010-08-01 Thread Heikki Linnakangas
On 27/07/10 16:12, Joshua Tolley wrote: My concern is that in a quorum system, if the quorum number is less than the total number of replicas, there's no way to know *which* replicas composed the quorum for any given transaction, so we can't know which servers to fail to if the master dies. In

Re: [HACKERS] Synchronous replication

2010-08-01 Thread Greg Stark
On Sun, Aug 1, 2010 at 7:11 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: In fact, it's possible for one standby to sync up to X, then disconnect and reconnect, and have the master count it second time in the quorum. Especially if the master doesn't notice that the standby

Re: [HACKERS] Synchronous replication

2010-08-01 Thread Robert Haas
On Sun, Aug 1, 2010 at 8:30 AM, Greg Stark gsst...@mit.edu wrote: On Sun, Aug 1, 2010 at 7:11 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: In fact, it's possible for one standby to sync up to X, then disconnect and reconnect, and have the master count it second time in the

Re: [HACKERS] Synchronous replication

2010-08-01 Thread Fujii Masao
On Sun, Aug 1, 2010 at 9:30 PM, Greg Stark gsst...@mit.edu wrote: This doesn't have to be done manually. Agreed, if we register standbys in the master. The streaming protocol could include the standby sending its system id to the master. The master could just keep a list of system ids with

Re: [HACKERS] Synchronous replication

2010-08-01 Thread Robert Haas
On Sun, Aug 1, 2010 at 10:08 PM, Fujii Masao masao.fu...@gmail.com wrote: On Sun, Aug 1, 2010 at 9:30 PM, Greg Stark gsst...@mit.edu wrote: This doesn't have to be done manually. Agreed, if we register standbys in the master. The streaming protocol could include the standby sending its

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Yeb Havinga
Fujii Masao wrote: On Mon, Jul 26, 2010 at 8:25 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jul 26, 2010 at 6:48 AM, Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: On 7/26/10 1:44 PM +0300, Fujii Masao wrote: On Mon, Jul 26, 2010 at 6:36 PM, Yeb

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Yeb Havinga
Joshua Tolley wrote: Perhaps I'm hijacking the wrong thread for this, but I wonder if the quorum idea is really the best thing for us. For reference: it appeared in a long thread a while ago http://archives.postgresql.org/pgsql-hackers/2010-05/msg01226.php. In short, there are three different

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Yeb Havinga
Fujii Masao wrote: The attached patch changes the backend so that it signals walsender to wake up from the sleep and send WAL immediately. It doesn't include any other synchronous replication stuff. Hello Fujii, I noted the changes in XlogSend where instead of *caughtup = true/false it now

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Fujii Masao
On Tue, Jul 27, 2010 at 7:39 PM, Yeb Havinga yebhavi...@gmail.com wrote: Fujii Masao wrote: The attached patch changes the backend so that it signals walsender to wake up from the sleep and send WAL immediately. It doesn't include any other synchronous replication stuff. Hello Fujii,

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Fujii Masao
On Tue, Jul 27, 2010 at 5:42 PM, Yeb Havinga yebhavi...@gmail.com wrote: I'd like to bring forward another suggestion (please tell me when it is becoming spam). My feeling about replication_mode as is, is that is says in the same parameter something about async or sync, as well as, if sync,

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Yeb Havinga
Fujii Masao wrote: I noted the changes in XlogSend where instead of *caughtup = true/false it now returns !MyWalSnd-sndrqst. That value is initialized to false in that procedure and it cannot be changed to true during execution of that procedure, or can it? That value is set to true in

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Joshua Tolley
On Tue, Jul 27, 2010 at 01:41:10PM +0900, Fujii Masao wrote: On Tue, Jul 27, 2010 at 12:36 PM, Joshua Tolley eggyk...@gmail.com wrote: Perhaps I'm hijacking the wrong thread for this, but I wonder if the quorum idea is really the best thing for us. I've been thinking about Oracle's way of

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Fujii Masao
On Tue, Jul 27, 2010 at 8:48 PM, Yeb Havinga yebhavi...@gmail.com wrote: Is there a reason not to send the signal in XlogFlush itself, so it would be called at CreateCheckPoint(), EndPrepare(), FlushBuffer(), RecordTransactionAbortPrepared(), RecordTransactionCommit(),

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Fujii Masao
On Tue, Jul 27, 2010 at 10:12 PM, Joshua Tolley eggyk...@gmail.com wrote: I don't think it can support the case you're interested in, though I'm not terribly expert on it. I'm definitely not arguing for the syntax Oracle uses, or something similar; I much prefer the flexibility we're proposing,

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Joshua Tolley
On Tue, Jul 27, 2010 at 10:53:45PM +0900, Fujii Masao wrote: On Tue, Jul 27, 2010 at 10:12 PM, Joshua Tolley eggyk...@gmail.com wrote: My concern is that in a quorum system, if the quorum number is less than the total number of replicas, there's no way to know *which* replicas composed the

Re: [HACKERS] Synchronous replication

2010-07-27 Thread Dimitri Fontaine
Le 27 juil. 2010 à 15:12, Joshua Tolley eggyk...@gmail.com a écrit : My concern is that in a quorum system, if the quorum number is less than the total number of replicas, there's no way to know *which* replicas composed the quorum for any given transaction, so we can't know which servers to

Re: [HACKERS] Synchronous replication

2010-07-26 Thread Fujii Masao
On Thu, Jul 22, 2010 at 5:37 PM, Yeb Havinga yebhavi...@gmail.com wrote: Fujii Masao wrote: How should the synchronous replication behave when the number of connected standby servers is less than quorum? 1. Ignore quorum. The current patch adopts this. If the ACKs from all   connected

Re: [HACKERS] Synchronous replication

2010-07-26 Thread Yeb Havinga
Fujii Masao wrote: Intuitively by looking at the enumeration of replication_mode I'd think that the sync standbys are all standby's that operate in a not async mode. That would be clearer with a boolean sync (or not) and for sync standbys the replication_mode specified. You mean that

Re: [HACKERS] Synchronous replication

2010-07-26 Thread Fujii Masao
On Mon, Jul 26, 2010 at 5:27 PM, Yeb Havinga yebhavi...@gmail.com wrote: Fujii Masao wrote: Intuitively by looking at the enumeration of replication_mode I'd think that the sync standbys are all standby's that operate in a not async mode. That would be clearer with a boolean sync (or not)

quorum commit Re: [HACKERS] Synchronous replication

2010-07-26 Thread Fujii Masao
On Thu, Jul 22, 2010 at 5:37 PM, Yeb Havinga yebhavi...@gmail.com wrote: Fujii Masao wrote: How should the synchronous replication behave when the number of connected standby servers is less than quorum? 1. Ignore quorum. The current patch adopts this. If the ACKs from all   connected

Re: quorum commit Re: [HACKERS] Synchronous replication

2010-07-26 Thread Yeb Havinga
Fujii Masao wrote: In the following case, how should quorum commit behave? 1. quorum_standbys = 2; there are three connected synchronous standbys 2. One standby sends the ACK back and fails 3. The ACK arrives from another standby 4. How should quorum commit behave? (a) Transaction commit

  1   2   3   >