Re: [HACKERS] is sync rep stalled?

2010-10-14 Thread Bruce Momjian
Heikki Linnakangas wrote: On 04.10.2010 10:49, Markus Wanner wrote: On 10/04/2010 09:18 AM, Heikki Linnakangas wrote: With 'replay' and hot standby combination, you'll want to set max_standby_archive_delay to a very low value, or a read-only query can cause master to stop processing

Re: [HACKERS] is sync rep stalled?

2010-10-06 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: I think the point here is that it's possible to have sync-rep configurations in which it's impossible to take a base backup. Sorry to be slow. I still don't understand that problem. I can understand why people want wait forever, but I can't understand when

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Heikki Linnakangas
On 04.10.2010 17:22, Fujii Masao wrote: If we make all the transactions wait until specified standbys have connected to the master, how do we take a base backup from the master for those standbys? We seem to be unable to do that because pg_start_backup also waits forever. Is this right? Hmm,

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Fujii Masao
On Tue, Oct 5, 2010 at 5:49 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 04.10.2010 17:22, Fujii Masao wrote: If we make all the transactions wait until specified standbys have connected to the master, how do we take a base backup from the master for those standbys? We

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Heikki Linnakangas
On 05.10.2010 12:47, Fujii Masao wrote: I have another question: when should the waiting transactions resume? It's a moment the standby has connected to the master? It's a moment the standby has caught up with the master? For no data loss, the latter seems to be required. Right? Yep. The

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Simon Riggs
On Tue, 2010-10-05 at 18:47 +0900, Fujii Masao wrote: On Tue, Oct 5, 2010 at 5:49 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 04.10.2010 17:22, Fujii Masao wrote: If we make all the transactions wait until specified standbys have connected to the master, how do

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Simon Riggs
On Fri, 2010-10-01 at 07:16 -0700, David Fetter wrote: On Fri, Oct 01, 2010 at 07:48:25PM +0900, Fujii Masao wrote: I proposed to implement the return-immediately at first because it doesn't require standby registration. But if many people think that the wait-forever is the core rather than

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Simon Riggs
On Fri, 2010-10-01 at 19:48 +0900, Fujii Masao wrote: My intention is to commit the core part of synchronous replication (which would be used for every use cases) at first. Then we can implement the feature for each use case. I completely agree that we should commit the core part of sync

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Aidan Van Dyk
On Mon, Oct 4, 2010 at 11:48 PM, Fujii Masao masao.fu...@gmail.com wrote: How can we take a base backup for that synchronous standby? You mean that we should disable the wait-forever option, start the master, take a base backup, shut down the master, enable the wait-forever option, start the

Re: [HACKERS] is sync rep stalled?

2010-10-05 Thread Fujii Masao
On Tue, Oct 5, 2010 at 8:25 PM, Simon Riggs si...@2ndquadrant.com wrote: Presumably we will have an option to *not* wait forever? So we would be able to set the option prior to running the base backup? So there isn't any need to do this rollback trick suggested. At the initial setup of the

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Markus Wanner
On 10/01/2010 05:06 PM, Dimitri Fontaine wrote: Wait forever can be done without standby registration, with quorum commit. Yeah, I also think the only reason for standby registration is ease of configuration (if at all). There's no technical requirement for standby registration, AFAICS. Or does

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Markus Wanner
On 10/04/2010 09:18 AM, Heikki Linnakangas wrote: Note that this assumes that you use the 'replay' synchronization level. In the weaker levels, read-only queries can always return stale data. I'm not too found of those various synchronization levels, but IIUC all other levels only allow a

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Markus Wanner
On 09/30/2010 04:54 PM, Yeb Havinga wrote: Heikki Linnakangas wrote: You do realize that to be able to guarantee zero data loss, the master will have to stop committing new transactions if the streaming stops for any reason, like a network glitch. Maybe that's a tradeoff you want, but I'm

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Heikki Linnakangas
On 04.10.2010 10:03, Markus Wanner wrote: On 09/30/2010 04:54 PM, Yeb Havinga wrote: Heikki Linnakangas wrote: You do realize that to be able to guarantee zero data loss, the master will have to stop committing new transactions if the streaming stops for any reason, like a network glitch.

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Heikki Linnakangas
On 04.10.2010 10:49, Markus Wanner wrote: On 10/04/2010 09:18 AM, Heikki Linnakangas wrote: With 'replay' and hot standby combination, you'll want to set max_standby_archive_delay to a very low value, or a read-only query can cause master to stop processing commits (or the standby to stop

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Markus Wanner
Hi, On 10/03/2010 05:52 AM, Josh Berkus wrote: (3) Invite people to comment on the wiki by leaving per-item comments and suggestions with their own names. Please keep discussions on the mailing list. On Wikis, those are very hard to follow (Date or From missing, no offline capabilities,

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Fujii Masao
On Fri, Oct 1, 2010 at 11:16 PM, David Fetter da...@fetter.org wrote: On Fri, Oct 01, 2010 at 07:48:25PM +0900, Fujii Masao wrote: I proposed to implement the return-immediately at first because it doesn't require standby registration. But if many people think that the wait-forever is the core

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Aidan Van Dyk
On Mon, Oct 4, 2010 at 10:22 AM, Fujii Masao masao.fu...@gmail.com wrote: I have one question for clarity: If we make all the transactions wait until specified standbys have connected to the master, how do we take a base backup from the master for those standbys? We seem to be unable to do

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Fujii Masao
On Tue, Oct 5, 2010 at 2:06 AM, Aidan Van Dyk ai...@highrise.ca wrote: On Mon, Oct 4, 2010 at 10:22 AM, Fujii Masao masao.fu...@gmail.com wrote: I have one question for clarity: If we make all the transactions wait until specified standbys have connected to the master, how do we take a base

Re: [HACKERS] is sync rep stalled?

2010-10-04 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes: On Tue, Oct 5, 2010 at 2:06 AM, Aidan Van Dyk ai...@highrise.ca wrote: I'm happy with the docs saying (maybe some what more politely):  Before configuring your master to be completly, wait-fully-synchronous, make sure you have a slave capable of being

Re: [HACKERS] is sync rep stalled?

2010-10-02 Thread Josh Berkus
On 09/30/2010 10:52 PM, Tom Lane wrote: IMHO we should push out the end date by at least a week to reflect the lack of time spent on the CF so far. I agree that we should postpone the end of the CF by one week to deal with the distractions people have had. --

Re: [HACKERS] is sync rep stalled?

2010-10-02 Thread Josh Berkus
What we actually need to do is talk and listen. I'd like to suggest that we have an online focus day (onlist) on Sync Rep on Oct 5 and maybe 6 as well?. Meeting in person is possible, but probably impractical. But a design sprint, not a code sprint. I'd suggest something even simpler: (1)

Re: [HACKERS] is sync rep stalled?

2010-10-01 Thread Fujii Masao
On Thu, Sep 30, 2010 at 3:09 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: * Support multiple standbys with various synchronization levels. Not required for that case. IMHO at least we'll still need to support asynchronous standbys in the same mix, that's an existing

Re: [HACKERS] is sync rep stalled?

2010-10-01 Thread David Fetter
On Fri, Oct 01, 2010 at 07:48:25PM +0900, Fujii Masao wrote: I proposed to implement the return-immediately at first because it doesn't require standby registration. But if many people think that the wait-forever is the core rather than the return-immediately, I'll follow them. We can

Re: [HACKERS] is sync rep stalled?

2010-10-01 Thread Dimitri Fontaine
Fujii Masao masao.fu...@gmail.com writes: I proposed to implement the return-immediately at first because it doesn't require standby registration. But if many people think that the wait-forever is the core rather than the return-immediately, I'll follow them. We can implement the

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Heikki Linnakangas
On 29.09.2010 10:56, Fujii Masao wrote: On Wed, Sep 29, 2010 at 11:47 AM, Robert Haasrobertmh...@gmail.com wrote: So we've got two patches that implement synchronous replication, and no agreement on which one, if either, should be committed. We have no agreement on how synchronous replication

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Simon Riggs
On Thu, 2010-09-30 at 09:09 +0300, Heikki Linnakangas wrote: On 29.09.2010 10:56, Fujii Masao wrote: On Wed, Sep 29, 2010 at 11:47 AM, Robert Haasrobertmh...@gmail.com wrote: This feature is important, and we need to get it done. How do we get the ball rolling again? Agreed. Actually,

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread David Fetter
On Thu, Sep 30, 2010 at 09:14:42AM +0100, Simon Riggs wrote: On Thu, 2010-09-30 at 09:09 +0300, Heikki Linnakangas wrote: On 29.09.2010 10:56, Fujii Masao wrote: On Wed, Sep 29, 2010 at 11:47 AM, Robert Haasrobertmh...@gmail.com wrote: This feature is important, and we need to get

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Tom Lane
David Fetter da...@fetter.org writes: On Thu, Sep 30, 2010 at 09:14:42AM +0100, Simon Riggs wrote: I don't see anything has stalled. I do. We're half way through this commitfest, so if no one's actually ready to commit one of the patches, I kinda have to bounce them both, at least to the

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Aidan Van Dyk
On Thu, Sep 30, 2010 at 2:09 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Agreed. Actually, given the lack of people jumping in and telling us what they'd like to do with the feature, maybe it's not that important after all. The basic features that I mean is for most

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread David Fetter
On Thu, Sep 30, 2010 at 09:52:46AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Thu, Sep 30, 2010 at 09:14:42AM +0100, Simon Riggs wrote: I don't see anything has stalled. I do. We're half way through this commitfest, so if no one's actually ready to commit one of

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Kevin Grittner
Aidan Van Dyk ai...@highrise.ca wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I'm sure there's several things you can accomplish with synchronous replication, perhaps you could describe what the important use case for you is? I'm looking for data durability, not

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Heikki Linnakangas
On 30.09.2010 17:09, Kevin Grittner wrote: Aidan Van Dykai...@highrise.ca wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com wrote: I'm sure there's several things you can accomplish with synchronous replication, perhaps you could describe what the important use case for you is?

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Yeb Havinga
Heikki Linnakangas wrote: On 30.09.2010 17:09, Kevin Grittner wrote: Aidan Van Dykai...@highrise.ca wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com wrote: I'm sure there's several things you can accomplish with synchronous replication, perhaps you could describe what the

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: You do realize that to be able to guarantee zero data loss, the master will have to stop committing new transactions if the streaming stops for any reason, like a network glitch. Maybe that's a tradeoff you want, but I'm asking

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Simon Riggs
On Thu, 2010-09-30 at 07:06 -0700, David Fetter wrote: On Thu, Sep 30, 2010 at 09:52:46AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Thu, Sep 30, 2010 at 09:14:42AM +0100, Simon Riggs wrote: I don't see anything has stalled. I do. We're half way through this

Re: [HACKERS] is sync rep stalled?

2010-09-30 Thread Robert Haas
On Thu, Sep 30, 2010 at 12:52 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-09-30 at 07:06 -0700, David Fetter wrote: On Thu, Sep 30, 2010 at 09:52:46AM -0400, Tom Lane wrote: David Fetter da...@fetter.org writes: On Thu, Sep 30, 2010 at 09:14:42AM +0100, Simon Riggs wrote: I

Re: [HACKERS] is sync rep stalled?

2010-09-29 Thread Fujii Masao
On Wed, Sep 29, 2010 at 11:47 AM, Robert Haas robertmh...@gmail.com wrote: So we've got two patches that implement synchronous replication, and no agreement on which one, if either, should be committed.  We have no agreement on how synchronous replication should be configured, and at most a

Re: [HACKERS] is sync rep stalled?

2010-09-29 Thread Robert Haas
On Wed, Sep 29, 2010 at 3:56 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 29, 2010 at 11:47 AM, Robert Haas robertmh...@gmail.com wrote: So we've got two patches that implement synchronous replication, and no agreement on which one, if either, should be committed.  We have no

[HACKERS] is sync rep stalled?

2010-09-28 Thread Robert Haas
So we've got two patches that implement synchronous replication, and no agreement on which one, if either, should be committed. We have no agreement on how synchronous replication should be configured, and at most a tenuous agreement that it should involve standby registration. This is bad.