Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-08 Thread Heikki Linnakangas
Fujii Masao wrote: On Sun, Feb 7, 2010 at 1:02 AM, Bruce Momjian br...@momjian.us wrote: src/backend/access/transam/xlog.c else { XLogRecPtr InvalidXLogRecPtr = {0, 0}; ControlFile-minRecoveryPoint = InvalidXLogRecPtr; } In my original patch, the above is for the problem

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-08 Thread Fujii Masao
On Mon, Feb 8, 2010 at 6:11 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I removed that. Thanks! It only makes a difference if you stop archive recovery, remove recovery.conf, and start up again, causing the server to do normal crash recovery. That's a don't do that

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-07 Thread Fujii Masao
On Sun, Feb 7, 2010 at 1:02 AM, Bruce Momjian br...@momjian.us wrote: src/backend/access/transam/xlog.c else {     XLogRecPtr      InvalidXLogRecPtr = {0, 0};     ControlFile-minRecoveryPoint = InvalidXLogRecPtr; } In my original patch, the above is for the problem discussed in

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-06 Thread Bruce Momjian
Fujii Masao wrote: On Mon, Jan 4, 2010 at 9:55 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Heikki Linnakangas wrote: Here's is modified patch that adds a new backupStartPoint field to pg_control for that + some other minor editorialization. I've committed this

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-01-04 Thread Heikki Linnakangas
Heikki Linnakangas wrote: Here's is modified patch that adds a new backupStartPoint field to pg_control for that + some other minor editorialization. I've committed this now. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-01-04 Thread Fujii Masao
On Mon, Jan 4, 2010 at 9:55 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Heikki Linnakangas wrote: Here's is modified patch that adds a new backupStartPoint field to pg_control for that + some other minor editorialization. I've committed this now. Thanks a lot!

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-30 Thread Heikki Linnakangas
Fujii Masao wrote: On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok. How about writing the history file in pg_stop_backup() for informational purposes only. Ie. never

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-30 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-12-23 at 12:50 +0200, Heikki Linnakangas wrote: I just realized that the current history file fails to recognize this scenario: 1. pg_start_backup() 2. cp -a $PGDATA data-backup 3. create data-backup/recovery.conf 4. postmaster -D data-backup That is,

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-30 Thread Simon Riggs
On Wed, 2009-12-30 at 15:31 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: The code has always been capable of starting without this, which was considered a feature to be able start from a hot copy. Why is that desirable? The system is in an inconsistent state. To force it, you can

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok. How about writing the history file in pg_stop_backup() for informational purposes only. Ie. never read it, but rely on the WAL records instead. Sounds good. I'll make such change as a

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Simon Riggs
On Wed, 2009-12-23 at 12:50 +0200, Heikki Linnakangas wrote: I just realized that the current history file fails to recognize this scenario: 1. pg_start_backup() 2. cp -a $PGDATA data-backup 3. create data-backup/recovery.conf 4. postmaster -D data-backup That is, starting postmaster

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok. How about writing the history file in pg_stop_backup() for informational purposes only. Ie. never read it, but rely on

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Thu, Dec 24, 2009 at 5:35 PM, Simon Riggs si...@2ndquadrant.com wrote: The code has always been capable of starting without this, which was considered a feature to be able start from a hot copy. I would like to do as you suggest, but it would remove the feature. This would be a great

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Thu, Dec 24, 2009 at 5:38 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok. How about writing the history file in

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-23 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I think we could get away without the backup history file altogether. Hmmm ... actually I was confusing these with timeline history files, which are definitely not something we can drop. You might be right that

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Thu, 2009-11-26 at 17:02 +0900, Fujii Masao wrote: On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: In current SR, since a backup history file is not replicated, the standby always starts an archive recovery without a

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Tue, 2009-12-22 at 14:40 +0900, Fujii Masao wrote: On Sat, Dec 19, 2009 at 1:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't think it's worthwhile to modify pg_stop_backup() like that. We should address the general problem. At the moment, you're fine if you

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 2:42 AM, Simon Riggs si...@2ndquadrant.com wrote: The backup history file is a slightly bit quirky way of doing things and was designed when the transfer mechanism was file-based. Why don't we just write a new xlog record that contains the information we need? Copy the

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Wed, 2009-12-23 at 03:28 +0900, Fujii Masao wrote: On Wed, Dec 23, 2009 at 2:42 AM, Simon Riggs si...@2ndquadrant.com wrote: The backup history file is a slightly bit quirky way of doing things and was designed when the transfer mechanism was file-based. Why don't we just write a new

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 2:49 AM, Simon Riggs si...@2ndquadrant.com wrote: (I don't fully understand your words above, sorry.) NM ;-) Possibly an easier way would be to have a size limit, not a number of segments. Something like max_reserved_wal = 240GB. We made that change to shared_buffers

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: The backup history file is a slightly bit quirky way of doing things and was designed when the transfer mechanism was file-based. Why don't we just write a new xlog record that contains the information we need? Certainly not. The history file is, in

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 3:41 AM, Simon Riggs si...@2ndquadrant.com wrote: This means that we can replace a backup history file with the corresponding xlog record. I think that we should simplify the code by making the replacement completely rather than just adding new xlog record. Thought?

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Tue, 2009-12-22 at 14:02 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The backup history file is a slightly bit quirky way of doing things and was designed when the transfer mechanism was file-based. Why don't we just write a new xlog record that contains the

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs si...@2ndquadrant.com wrote: It's needed to find the place where the backup stopped, so it defines the safe stopping point. We could easily pass that info via WAL, when streaming. It doesn't actually matter until we try to failover. Right. And, it's

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Wed, 2009-12-23 at 04:15 +0900, Fujii Masao wrote: On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs si...@2ndquadrant.com wrote: It's needed to find the place where the backup stopped, so it defines the safe stopping point. We could easily pass that info via WAL, when streaming. It doesn't

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes: On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs si...@2ndquadrant.com wrote: It's needed to find the place where the backup stopped, so it defines the safe stopping point. We could easily pass that info via WAL, when streaming. It doesn't actually matter

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Heikki Linnakangas
Simon Riggs wrote: On Wed, 2009-12-23 at 04:15 +0900, Fujii Masao wrote: On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs si...@2ndquadrant.com wrote: It's needed to find the place where the backup stopped, so it defines the safe stopping point. We could easily pass that info via WAL, when

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Heikki Linnakangas
Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: The backup history file is a slightly bit quirky way of doing things and was designed when the transfer mechanism was file-based. Why don't we just write a new xlog record that contains the information we need? Certainly not. The

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I think we could get away without the backup history file altogether. Hmmm ... actually I was confusing these with timeline history files, which are definitely not something we can drop. You might be right that the backup history

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Aidan Van Dyk
* Heikki Linnakangas heikki.linnakan...@enterprisedb.com [091222 15:47]: I was just having a chat with Magnus this morning, and he asked if the current patch already provides or if it would be possible to write a stand-alone utility to connect to a master and stream WAL files to an archive

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Tue, 2009-12-22 at 22:46 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: On Wed, 2009-12-23 at 04:15 +0900, Fujii Masao wrote: On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs si...@2ndquadrant.com wrote: It's needed to find the place where the backup stopped, so it defines the safe

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-21 Thread Fujii Masao
On Sat, Dec 19, 2009 at 1:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't think it's worthwhile to modify pg_stop_backup() like that. We should address the general problem. At the moment, you're fine if you also configure WAL archiving and log file shipping, but it

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-19 Thread Greg Smith
Robert Haas wrote: I think (as I did/do with Hot Standby) that the most important thing here is to get to a point where we have a reasonably good feature that is of some use, and commit it. It will probably have some annoying limitations; we can remove those later. I have a feel that what we

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Heikki Linnakangas
Fujii Masao wrote: pg_stop_backup deletes the previous backup history file from pg_xlog. So replication of a backup history file would fail if just one new online-backup is caused after the base-backup for the standby is taken. This is too aggressive deletion policy for Streaming Replication,

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Robert Haas
On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Or some way for to register the standby with the master so that the master knows it's out there, and still needs the logs, even when it's not connected. That is the real answer, I think. ...Robert

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Florian Pflug
On 18.12.09 17:05 , Robert Haas wrote: On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Or some way for to register the standby with the master so that the master knows it's out there, and still needs the logs, even when it's not connected. That

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Robert Haas
On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug fgp.phlo@gmail.com wrote: On 18.12.09 17:05 , Robert Haas wrote: On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: Or some way for to register the standby with the master so that the master

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Heikki Linnakangas
Robert Haas wrote: On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug fgp.phlo@gmail.com wrote: On 18.12.09 17:05 , Robert Haas wrote: On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Or some way for to register the standby with the master so

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Dimitri Fontaine
Hi, Le 18 déc. 2009 à 19:21, Heikki Linnakangas a écrit : On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug fgp.phlo@gmail.com wrote: It'd prefer if the slave could automatically fetch a new base backup if it falls behind too far to catch up with the available logs. That way, old logs

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Alvaro Herrera
Dimitri Fontaine escribió: Well I did propose to consider a state machine with clear transition for such problems, a while ago, and I think my remarks still do apply: http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg131511.html Sorry for non archives.postgresql.org link,

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-04 Thread Fujii Masao
Hi, On Thu, Nov 26, 2009 at 5:20 PM, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Nov 26, 2009 at 5:17 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Yeah, that needs to be addressed regardless of HS, because you can otherwise start up (= fail over to) the standby too

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-26 Thread Fujii Masao
On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: In current SR, since a backup history file is not replicated, the standby always starts an archive recovery without a backup history file, and a wrong minRecoveryPoint might be

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-26 Thread Heikki Linnakangas
Fujii Masao wrote: On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: In current SR, since a backup history file is not replicated, the standby always starts an archive recovery without a backup history file, and a wrong

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-26 Thread Fujii Masao
On Thu, Nov 26, 2009 at 5:17 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Yeah, that needs to be addressed regardless of HS, because you can otherwise start up (= fail over to) the standby too early, before the minimum recovery point has been reached. Okey, I address that

[HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-25 Thread Fujii Masao
Hi, In current SR, since a backup history file is not replicated, the standby always starts an archive recovery without a backup history file, and a wrong minRecoveryPoint might be used. This is not a problem for SR itself, but would cause trouble when SR cooperates with Hot Standby. HS begins

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-25 Thread Heikki Linnakangas
Fujii Masao wrote: In current SR, since a backup history file is not replicated, the standby always starts an archive recovery without a backup history file, and a wrong minRecoveryPoint might be used. This is not a problem for SR itself, but would cause trouble when SR cooperates with Hot