Re: [HACKERS] Streaming replication, and walsender during recovery

2010-03-16 Thread Heikki Linnakangas
Fujii Masao wrote: On Mon, Jan 18, 2010 at 2:19 PM, Fujii Masao masao.fu...@gmail.com wrote: When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following errors, and a cascaded standby didn't start replication. ERROR: timeline 0

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-03-16 Thread Fujii Masao
On Tue, Mar 16, 2010 at 6:11 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Commmitted. I edited the message and error code a bit: ereport(FATAL,        (errcode(ERRCODE_CANNOT_CONNECT_NOW),         errmsg(recovery is still in progress, can't accept WAL streaming

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-02-17 Thread Fujii Masao
On Mon, Jan 18, 2010 at 2:19 PM, Fujii Masao masao.fu...@gmail.com wrote: When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following errors, and a cascaded standby didn't start replication.  ERROR:  timeline 0 of the primary does

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 09:49 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2010-01-28 at 21:00 +0200, Heikki Linnakangas wrote: I think it is a pretty important safety feature that we keep all the WAL around that's needed to recover the standby. To avoid out-of-disk-space

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Fujii Masao
On Fri, Jan 29, 2010 at 4:13 AM, Simon Riggs si...@2ndquadrant.com wrote: Hmm, I'm sorry but that's bogus. Retaining so much WAL that we are strongly in danger of blowing disk space is not what I would call a safety feature. Since there is no way to control or restrain the number of files for

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Simon Riggs
On Fri, 2010-01-29 at 17:31 +0900, Fujii Masao wrote: On Fri, Jan 29, 2010 at 4:13 AM, Simon Riggs si...@2ndquadrant.com wrote: Hmm, I'm sorry but that's bogus. Retaining so much WAL that we are strongly in danger of blowing disk space is not what I would call a safety feature. Since there

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-29 Thread Fujii Masao
On Fri, Jan 29, 2010 at 5:41 PM, Simon Riggs si...@2ndquadrant.com wrote: To improve the situation, I think that we need to use checkpoint_segment/timeout as a trigger of restartpoint, regardless of the checkpoint record. Though I'm not sure that is possible and should be included in v9.0.

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Fujii Masao
On Thu, Jan 28, 2010 at 4:47 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think there's a race condition at the end of recovery. When the shutdown checkpoint is written, with new TLI, doesn't a cascading walsender try to send that to the standby as soon as it's flushed to

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Fujii Masao wrote: On Thu, Jan 28, 2010 at 4:47 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I think there's a race condition at the end of recovery. When the shutdown checkpoint is written, with new TLI, doesn't a cascading walsender try to send that to the standby as

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Fujii Masao
On Thu, Jan 28, 2010 at 7:43 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: How about restoring/streaming files to a new directory, say pg_xlog/restored/, with the real filenames? At least in standby_mode, probably best to keep the current behavior in PITR. That would feel

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes: How about just making a restore_command copy the WAL files as the normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? Though we need to worry about deleting them, we can easily leave the task to the bgwriter. The reason for doing it that way

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 10:48 -0500, Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: How about just making a restore_command copy the WAL files as the normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? Though we need to worry about deleting them, we can easily leave the

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: I'm keen to allow cascading in 9.0. If you pull both synch rep and cascading you're not offering much that isn't already there. FWIW, I don't agree with that prioritization in the least. Cascading is something we could leave till 9.1, or even later,

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Robert Haas
On Thu, Jan 28, 2010 at 11:41 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: I'm keen to allow cascading in 9.0. If you pull both synch rep and cascading you're not offering much that isn't already there. FWIW, I don't agree with that prioritization in the

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 12:09 -0500, Robert Haas wrote: I agree. According to http://wiki.postgresql.org/wiki/Hot_Standby_TODO , the only must-fix issues that remain prior to beta are (1) implementing the new VACUUM FULL for system relations, and (2) some documentation improvements. It's a

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 11:41 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: I'm keen to allow cascading in 9.0. If you pull both synch rep and cascading you're not offering much that isn't already there. FWIW, I don't agree with that prioritization in the least. Cascading

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On Thu, 2010-01-28 at 11:41 -0500, Tom Lane wrote: FWIW, I don't agree with that prioritization in the least. Cascading is something we could leave till 9.1, or even later, and Not what you said just a few days ago. Me? I don't recall having said

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 13:05 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2010-01-28 at 11:41 -0500, Tom Lane wrote: FWIW, I don't agree with that prioritization in the least. Cascading is something we could leave till 9.1, or even later, and Not what you

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Greg Smith
Tom Lane wrote: (Anyway, the argument that it's important for performance is pure speculation AFAIK, untainted by any actual measurements. Given the lack of optimization of WAL replay, it seems entirely possible that the last thing you want to burden a slave with is sourcing data to more

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: How about just making a restore_command copy the WAL files as the normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? Though we need to worry about deleting them, we can easily leave the task to the bgwriter. The reason

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Joshua D. Drake
On Thu, 2010-01-28 at 13:05 -0500, Tom Lane wrote: I'm a little worried the feature set of streaming rep isn't any better than what we have already. Nonsense. Getting rid of the WAL-segment-based shipping delays is a quantum improvement --- it means we actually have something approaching

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: I'm a little worried the feature set of streaming rep isn't any better than what we have already. Huh? Are you thinking of the Record-based Log Shipping described in the manual, using a program to poll pg_xlogfile_name_offset() in a tight loop, as a replacement for streaming

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 20:49 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I'm a little worried the feature set of streaming rep isn't any better than what we have already. Huh? Are you thinking of the Record-based Log Shipping described in the manual, using a program to poll

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2010-01-28 at 10:48 -0500, Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: How about just making a restore_command copy the WAL files as the normal one (e.g., ...) instead of a pg_xlog/RECOVERYXLOG? Though we need to worry about deleting them, we can

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Simon Riggs
On Thu, 2010-01-28 at 21:00 +0200, Heikki Linnakangas wrote: However, since not every checkpoint is a restartpoint we might easily end up with significantly more WAL files on the standby than would normally be there when it would be a primary. Not sure if that is an issue in this case, but

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Joshua D. Drake wrote: ...if with SR the entire log must be written before it streams to the slaves. No. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Josh Berkus
Guys, Hmm, I'm sorry but that's bogus. Retaining so much WAL that we are strongly in danger of blowing disk space is not what I would call a safety feature. Since there is no way to control or restrain the number of files for certain, that approach seems fatally flawed. Reducing

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-28 Thread Heikki Linnakangas
Simon Riggs wrote: On Thu, 2010-01-28 at 21:00 +0200, Heikki Linnakangas wrote: I think it is a pretty important safety feature that we keep all the WAL around that's needed to recover the standby. To avoid out-of-disk-space situation, it's probably enough in practice to set

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-27 Thread Heikki Linnakangas
Fujii Masao wrote: OK. Here is the patch which supports a walsender process during recovery; * Change walsender so as to send the WAL written by the walreceiver if it has been started during recovery. * Kill the walsenders started during recovery at the end of recovery because

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-20 Thread Fujii Masao
On Tue, Jan 19, 2010 at 4:41 PM, Simon Riggs si...@2ndquadrant.com wrote: It would be useful to have the ps display differentiate between multiple walsenders, and in this case have it indicate cascading also. Since a normal walsender and a cascading one will not be running at the same time, I

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes: When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following errors, and a cascaded standby didn't start replication. ERROR: timeline 0 of the primary does not match recovery target

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Simon Riggs
On Mon, 2010-01-18 at 09:31 -0500, Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following errors, and a cascaded standby didn't start replication. ERROR:

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Fujii Masao
On Mon, Jan 18, 2010 at 11:42 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2010-01-18 at 09:31 -0500, Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following

Re: [HACKERS] Streaming replication, and walsender during recovery

2010-01-18 Thread Simon Riggs
On Tue, 2010-01-19 at 15:04 +0900, Fujii Masao wrote: There is an ERROR, but no problem AFAICS. The tli isn't set until end of recovery because it doesn't need to have been set yet. That shouldn't prevent retrieving WAL data. OK. Here is the patch which supports a walsender process

[HACKERS] Streaming replication, and walsender during recovery

2010-01-17 Thread Fujii Masao
Hi, When I configured a cascaded standby (i.e, made the additional standby server connect to the standby), I got the following errors, and a cascaded standby didn't start replication. ERROR: timeline 0 of the primary does not match recovery target timeline 1 I didn't care about that case so