Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-08 Thread Fujii Masao
On Mon, Feb 8, 2010 at 6:11 PM, Heikki Linnakangas wrote: > I removed that. Thanks! > It only makes a difference if you stop archive recovery, remove > recovery.conf, and start up again, causing the server to do normal crash > recovery. That's a "don't do that" scenario, but it seems better to n

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-08 Thread Heikki Linnakangas
Fujii Masao wrote: > On Sun, Feb 7, 2010 at 1:02 AM, Bruce Momjian wrote: >>> src/backend/access/transam/xlog.c else { XLogRecPtr InvalidXLogRecPtr = {0, 0}; ControlFile->minRecoveryPoint = InvalidXLogRecPtr; } >>> In my original patch, the above is for the pr

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-07 Thread Fujii Masao
On Sun, Feb 7, 2010 at 1:02 AM, Bruce Momjian wrote: >> src/backend/access/transam/xlog.c >> > else >> > { >> >     XLogRecPtr      InvalidXLogRecPtr = {0, 0}; >> >     ControlFile->minRecoveryPoint = InvalidXLogRecPtr; >> > } >> >> In my original patch, the above is for the problem discussed in >

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-02-06 Thread Bruce Momjian
Fujii Masao wrote: > On Mon, Jan 4, 2010 at 9:55 PM, Heikki Linnakangas > wrote: > > Heikki Linnakangas wrote: > >> Here's is modified patch that adds a new backupStartPoint field to > >> pg_control for that + some other minor editorialization. > > > > I've committed this now. > > Thanks a lot! >

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-01-04 Thread Fujii Masao
On Mon, Jan 4, 2010 at 9:55 PM, Heikki Linnakangas wrote: > Heikki Linnakangas wrote: >> Here's is modified patch that adds a new backupStartPoint field to >> pg_control for that + some other minor editorialization. > > I've committed this now. Thanks a lot! src/backend/access/transam/xlog.c > e

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2010-01-04 Thread Heikki Linnakangas
Heikki Linnakangas wrote: > Here's is modified patch that adds a new backupStartPoint field to > pg_control for that + some other minor editorialization. I've committed this now. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-30 Thread Simon Riggs
On Wed, 2009-12-30 at 15:31 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > The code has always been capable of starting without this, which was > > considered a feature to be able start from a hot copy. > > Why is that desirable? The system is in an inconsistent state. To force > it, yo

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-30 Thread Heikki Linnakangas
Simon Riggs wrote: > On Wed, 2009-12-23 at 12:50 +0200, Heikki Linnakangas wrote: > >> I just realized that the current history file fails to recognize this >> scenario: >> >> 1. pg_start_backup() >> 2. cp -a $PGDATA data-backup >> 3. create data-backup/recovery.conf >> 4. postmaster -D data-backu

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-30 Thread Heikki Linnakangas
Fujii Masao wrote: > On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao wrote: >> On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas >> wrote: >>> Ok. How about writing the history file in pg_stop_backup() for >>> informational purposes only. Ie. never read it, but rely on the WAL >>> records instead.

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Thu, Dec 24, 2009 at 5:38 PM, Fujii Masao wrote: > On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao wrote: >> On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas >> wrote: >>> Ok. How about writing the history file in pg_stop_backup() for >>> informational purposes only. Ie. never read it, but re

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Thu, Dec 24, 2009 at 5:35 PM, Simon Riggs wrote: > The code has always been capable of starting without this, which was > considered a feature to be able start from a hot copy. I would like to > do as you suggest, but it would remove the feature. This would be a > great example of why I don't w

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Thu, Dec 24, 2009 at 1:39 PM, Fujii Masao wrote: > On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas > wrote: >> Ok. How about writing the history file in pg_stop_backup() for >> informational purposes only. Ie. never read it, but rely on the WAL >> records instead. > > Sounds good. I'll mak

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Simon Riggs
On Wed, 2009-12-23 at 12:50 +0200, Heikki Linnakangas wrote: > I just realized that the current history file fails to recognize this > scenario: > > 1. pg_start_backup() > 2. cp -a $PGDATA data-backup > 3. create data-backup/recovery.conf > 4. postmaster -D data-backup > > That is, starting post

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-24 Thread Fujii Masao
On Wed, Dec 23, 2009 at 7:50 PM, Heikki Linnakangas wrote: > Ok. How about writing the history file in pg_stop_backup() for > informational purposes only. Ie. never read it, but rely on the WAL > records instead. Sounds good. I'll make such change as a self-contained patch. > I just realized tha

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-23 Thread Heikki Linnakangas
Tom Lane wrote: > Heikki Linnakangas writes: >> I think we could get away without the backup history file altogether. > > Hmmm ... actually I was confusing these with timeline history files, > which are definitely not something we can drop. You might be right > that the backup history file could

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Tue, 2009-12-22 at 22:46 +0200, Heikki Linnakangas wrote: > Simon Riggs wrote: > > On Wed, 2009-12-23 at 04:15 +0900, Fujii Masao wrote: > >> On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs wrote: > >>> It's needed to find the place where the backup stopped, so it defines > >>> the safe stopping p

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Aidan Van Dyk
* Heikki Linnakangas [091222 15:47]: > I was just having a chat with Magnus this morning, and he asked if the > current patch already provides or if it would be possible to write a > stand-alone utility to connect to a master and stream WAL files to an > archive directory, without setting up a fu

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Tom Lane
Heikki Linnakangas writes: > I think we could get away without the backup history file altogether. Hmmm ... actually I was confusing these with timeline history files, which are definitely not something we can drop. You might be right that the backup history file could be part of WAL instead. O

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Heikki Linnakangas
Tom Lane wrote: > Simon Riggs writes: >> The backup history file is a slightly bit quirky way of doing things and >> was designed when the transfer mechanism was file-based. > >> Why don't we just write a new xlog record that contains the information >> we need? > > Certainly not. The history f

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Heikki Linnakangas
Simon Riggs wrote: > On Wed, 2009-12-23 at 04:15 +0900, Fujii Masao wrote: >> On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs wrote: >>> It's needed to find the place where the backup stopped, so it defines >>> the safe stopping point. We could easily pass that info via WAL, when >>> streaming. It do

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Tom Lane
Fujii Masao writes: > On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs wrote: >> It's needed to find the place where the backup stopped, so it defines >> the safe stopping point. We could easily pass that info via WAL, when >> streaming. It doesn't actually matter until we try to failover. > Right.

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Wed, 2009-12-23 at 04:15 +0900, Fujii Masao wrote: > On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs wrote: > > It's needed to find the place where the backup stopped, so it defines > > the safe stopping point. We could easily pass that info via WAL, when > > streaming. It doesn't actually matter

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 4:09 AM, Simon Riggs wrote: > It's needed to find the place where the backup stopped, so it defines > the safe stopping point. We could easily pass that info via WAL, when > streaming. It doesn't actually matter until we try to failover. Right. And, it's also needed to coo

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Tue, 2009-12-22 at 14:02 -0500, Tom Lane wrote: > Simon Riggs writes: > > The backup history file is a slightly bit quirky way of doing things and > > was designed when the transfer mechanism was file-based. > > > Why don't we just write a new xlog record that contains the information > > we n

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 3:41 AM, Simon Riggs wrote: >> This means that we can replace a backup history file with the corresponding >> xlog record. I think that we should simplify the code by making the >> replacement >> completely rather than just adding new xlog record. Thought? > > We can't do

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Tom Lane
Simon Riggs writes: > The backup history file is a slightly bit quirky way of doing things and > was designed when the transfer mechanism was file-based. > Why don't we just write a new xlog record that contains the information > we need? Certainly not. The history file is, in the restore-from-

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 2:49 AM, Simon Riggs wrote: > (I don't fully understand your words above, sorry.) NM ;-) > Possibly an easier way would be to have a size limit, not a number of > segments. Something like max_reserved_wal = 240GB. We made that change > to shared_buffers a few years back a

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Wed, 2009-12-23 at 03:28 +0900, Fujii Masao wrote: > On Wed, Dec 23, 2009 at 2:42 AM, Simon Riggs wrote: > > The backup history file is a slightly bit quirky way of doing things and > > was designed when the transfer mechanism was file-based. > > > > Why don't we just write a new xlog record th

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Fujii Masao
On Wed, Dec 23, 2009 at 2:42 AM, Simon Riggs wrote: > The backup history file is a slightly bit quirky way of doing things and > was designed when the transfer mechanism was file-based. > > Why don't we just write a new xlog record that contains the information > we need? Copy the contents of the

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Tue, 2009-12-22 at 14:40 +0900, Fujii Masao wrote: > On Sat, Dec 19, 2009 at 1:03 AM, Heikki Linnakangas > wrote: > > I don't think it's worthwhile to modify pg_stop_backup() like that. We > > should address the general problem. At the moment, you're fine if you > > also configure WAL archiving

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-22 Thread Simon Riggs
On Thu, 2009-11-26 at 17:02 +0900, Fujii Masao wrote: > On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas > wrote: > > Fujii Masao wrote: > >> In current SR, since a backup history file is not replicated, > >> the standby always starts an archive recovery without a backup > >> history file, and

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-21 Thread Fujii Masao
On Sat, Dec 19, 2009 at 1:03 AM, Heikki Linnakangas wrote: > I don't think it's worthwhile to modify pg_stop_backup() like that. We > should address the general problem. At the moment, you're fine if you > also configure WAL archiving and log file shipping, but it would be nice > to have some simp

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-19 Thread Greg Smith
Robert Haas wrote: I think (as I did/do with Hot Standby) that the most important thing here is to get to a point where we have a reasonably good feature that is of some use, and commit it. It will probably have some annoying limitations; we can remove those later. I have a feel that what we ha

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Alvaro Herrera
Dimitri Fontaine escribió: > Well I did propose to consider a state machine with clear transition for such > problems, a while ago, and I think my remarks still do apply: > http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg131511.html > > Sorry for non archives.postgresql.org link, c

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Dimitri Fontaine
Hi, Le 18 déc. 2009 à 19:21, Heikki Linnakangas a écrit : > On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug > wrote: >>> It'd prefer if the slave could automatically fetch a new base backup if it >>> falls behind too far to catch up with the available logs. That way, old logs >>> don't start pil

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Heikki Linnakangas
Robert Haas wrote: > On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug > wrote: >> On 18.12.09 17:05 , Robert Haas wrote: >>> On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas >>> wrote: Or some way for to register the standby with the master so that the master knows it's out there,

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Robert Haas
On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug wrote: > On 18.12.09 17:05 , Robert Haas wrote: >> >> On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas >>  wrote: >>> >>> Or some way for to register the standby with the master so that >>> the master knows it's out there, and still needs the lo

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Florian Pflug
On 18.12.09 17:05 , Robert Haas wrote: On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas wrote: Or some way for to register the standby with the master so that the master knows it's out there, and still needs the logs, even when it's not connected. That is the real answer, I think. It'd

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Robert Haas
On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas wrote: > Or some way for to register the standby with the master so that > the master knows it's out there, and still needs the logs, even when > it's not connected. That is the real answer, I think. ...Robert -- Sent via pgsql-hackers maili

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-18 Thread Heikki Linnakangas
Fujii Masao wrote: > pg_stop_backup deletes the previous backup history file from pg_xlog. > So replication of a backup history file would fail if just one new > online-backup is caused after the base-backup for the standby is taken. > This is too aggressive deletion policy for Streaming Replicatio

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-12-04 Thread Fujii Masao
Hi, On Thu, Nov 26, 2009 at 5:20 PM, Fujii Masao wrote: > On Thu, Nov 26, 2009 at 5:17 PM, Heikki Linnakangas > wrote: >> Yeah, that needs to be addressed regardless of HS, because you can >> otherwise start up (= fail over to) the standby too early, before the >> minimum recovery point has been

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-26 Thread Fujii Masao
On Thu, Nov 26, 2009 at 5:17 PM, Heikki Linnakangas wrote: > Yeah, that needs to be addressed regardless of HS, because you can > otherwise start up (= fail over to) the standby too early, before the > minimum recovery point has been reached. Okey, I address that ASAP. Regards, -- Fujii Masao

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-26 Thread Heikki Linnakangas
Fujii Masao wrote: > On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas > wrote: >> Fujii Masao wrote: >>> In current SR, since a backup history file is not replicated, >>> the standby always starts an archive recovery without a backup >>> history file, and a wrong minRecoveryPoint might be used.

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-26 Thread Fujii Masao
On Thu, Nov 26, 2009 at 4:55 PM, Heikki Linnakangas wrote: > Fujii Masao wrote: >> In current SR, since a backup history file is not replicated, >> the standby always starts an archive recovery without a backup >> history file, and a wrong minRecoveryPoint might be used. This >> is not a problem f

Re: [HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-25 Thread Heikki Linnakangas
Fujii Masao wrote: > In current SR, since a backup history file is not replicated, > the standby always starts an archive recovery without a backup > history file, and a wrong minRecoveryPoint might be used. This > is not a problem for SR itself, but would cause trouble when > SR cooperates with Ho

[HACKERS] Backup history file should be replicated in Streaming Replication?

2009-11-25 Thread Fujii Masao
Hi, In current SR, since a backup history file is not replicated, the standby always starts an archive recovery without a backup history file, and a wrong minRecoveryPoint might be used. This is not a problem for SR itself, but would cause trouble when SR cooperates with Hot Standby. HS begins ac