[HACKERS] Online base backup from the hot-standby

2011-05-27 Thread Jun Ishiduka
Hi I would like to develop function for 'Online base backup from the hot-standby' in PostgreSQL 9.2. Todo : Allow hot file system backups on standby servers (http://wiki.postgresql.org/wiki/Todo) [GOAL] * Make pg_basebackup to execute to the hot-standby server and acquire

Re: [HACKERS] Online base backup from the hot-standby

2011-05-30 Thread Jun Ishiduka
I don't much like that approach. The standby would need to be able to write the backup history file to the archive at the end of backup, and we'd have to reintroduce the code to fetch it from archive and, when streaming, from the master. At the moment, the archiver doesn't even run in the

Re: [HACKERS] Online base backup from the hot-standby

2011-06-02 Thread Jun Ishiduka
I don't much like that approach. The standby would need to be able to write the backup history file to the archive at the end of backup, and we'd have to reintroduce the code to fetch it from archive and, when streaming, from the master. At the moment, the archiver doesn't even run in the

[HACKERS] Online base backup from the hot-standby

2011-06-14 Thread Jun Ishiduka
( Quotation from http://archives.postgresql.org/pgsql-hackers/2011-05/msg01396.php ) STEP1: Make startup process to acquire backup-end-position from not only backup-end record but also backup-history-file . * startup process allows to acquire backup-end-position

Re: [HACKERS] Online base backup from the hot-standby

2011-06-14 Thread Jun Ishiduka
I still think that's headed in the wrong direction. (http://archives.postgresql.org/pgsql-hackers/2011-05/msg01405.php) Please check these mails, and teach the reason for content of the wrong direction. (http://archives.postgresql.org/pgsql-hackers/2011-06/msg00209.php)

Re: [HACKERS] Online base backup from the hot-standby

2011-06-23 Thread Jun Ishiduka
1) Today you can do a backup by just calling pg_start_backup('x'); copy the data directory and pg_stop_backup(); You do not need to use pg_basebackup to create a backup. The solution you are proposing would require pg_basebackup to be used to build backups from standby servers. YES. 2)

Re: [HACKERS] Online base backup from the hot-standby

2011-06-23 Thread Jun Ishiduka
What I think he is proposing would not require using pg_stop_backup() but you could also modify pg_stop_back() to work as well. What do you think of this idea? Do you see how the patch can be reworked to accomplish this? The logic that not use pg_stop_backup() would be difficult, because

Re: [HACKERS] Online base backup from the hot-standby

2011-06-27 Thread Jun Ishiduka
Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to accomplish base backups from standby servers? If not what changes do you think should be made? I reconsider the way to not use pg_stop_backup(). Process of online base

Re: [HACKERS] Online base backup from the hot-standby

2011-06-30 Thread Jun Ishiduka
Process of online base backup on standby server: 1. pg_start_backup('x'); 2. copy the data directory 3. copy *pg_control* Who deletes the backup_label file created by pg_start_backup()? Isn't pg_stop_backup() required to do that? You need it to take the system out of

Re: [HACKERS] Online base backup from the hot-standby

2011-06-30 Thread Jun Ishiduka
On this commitfest, the goal of the patch is to be able to be recovered using Minimum recovery ending location in the control file. Done. Regards. Jun Ishizuka NTT Software Corporation TEL:045-317-7018 E-Mail: ishizuka@po.ntts.co.jp

Re: [HACKERS] Online base backup from the hot-standby

2011-07-04 Thread Jun Ishiduka
When the standby restarts after it crashes during recovery, it always checks whether recovery has reached the backup end location by using minRecoveryPoint even though the standby doesn't start from the backup. This looks odd. Certainly. But, in this case, the state before recovery starts

Re: [HACKERS] Online base backup from the hot-standby

2011-07-05 Thread Jun Ishiduka
What about using backupStartPoint to check whether this recovery started from the backup or not? No, postgres can check whether this recovery started from the backup or not, but can not check whether standby server or master (got backup from). Once recovery started, backupStartPoint is

Re: [HACKERS] Online base backup from the hot-standby

2011-07-07 Thread Jun Ishiduka
As you proposed, adding new field which stores the backup end location taken from minRecoveryPoint, into pg_control sounds good idea. Update patch. Regards. Jun Ishizuka NTT Software Corporation TEL:045-317-7018 E-Mail: ishizuka@po.ntts.co.jp

Re: [HACKERS] Online base backup from the hot-standby

2011-07-12 Thread Jun Ishiduka
This version of the patch adds a field into pg_controldata that tries to store the source of the base backup while in recovery mode. I think your ultimate goal with this patch is to be able to take a backup of a running hot-standby slave and recover it as another instance. This patch seems

[HACKERS] Online base backup from the hot-standby

2011-08-05 Thread Jun Ishiduka
I will provide a patch which can exeute pg_start/stop_backup including to solve above comment and conditions in next stage. Then please review. done. * Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of the data dir. 3. Call pg_stop_backup() on the standby. 4. Copy

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Jun Ishiduka
* Not correspond yet * full_page_write = off - If the primary is full_page_write = off, archive recovery may not act normally. Therefore the standby may need to check whether full_page_write = off to WAL. Isn't having a standby make the full_page_write = on in

Re: [HACKERS] Online base backup from the hot-standby

2011-08-16 Thread Jun Ishiduka
* Not correspond yet ?* full_page_write = off ? ?- If the primary is full_page_write = off, archive recovery may not act ? ? ? normally. Therefore the standby may need to check whether full_page_write ? ? ? = off to WAL. Isn't having a standby make the

Re: [HACKERS] Online base backup from the hot-standby

2011-08-16 Thread Jun Ishiduka
* Not correspond yet ?* full_page_write = off ? ?- If the primary is full_page_write = off, archive recovery may not act ? ? ? normally. Therefore the standby may need to check whether full_page_write ? ? ? = off to WAL. Isn't having a standby make the

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Jun Ishiduka
* Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of the data dir. 3. Call pg_stop_backup() on the standby. 4. Copy the control file on the standby to the backup. 5. Check whether the control file is status during hot standby with pg_controldata. ? - If

Re: [HACKERS] Online base backup from the hot-standby

2011-09-13 Thread Jun Ishiduka
Update patch. Changes: * set 'on' full_page_writes by user (in document) * read FROM: XX in backup_label (in xlog.c) * check status when pg_stop_backup is executed (in xlog.c) Hi, Created a patch in response to comments. * Procedure 1. Call pg_start_backup('x') on hot standby. 2.

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Jun Ishiduka
Attached is the updated version of the patch. I refactored the code, fixed some bugs, added lots of source code comments, improved the document, but didn't change the basic design. Please check this patch, and let's use this patch as the base if you agree with that. Thanks for update patch.

Re: [HACKERS] Online base backup from the hot-standby

2011-10-09 Thread Jun Ishiduka
I created a patch corresponding FPW. Fujii's patch (ver 9) is based. Manage own FPW in shared-memory (on master) * startup and walwriter process update it. startup initializes it after REDO. walwriter updates it when started or received SIGHUP. Insert WAL including a value of current

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
I can't see a reason why we would use a new WAL record for this, rather than modify the XLOG_PARAMETER_CHANGE record type which was created for a very similar reason. The code would be much simpler if we just extend XLOG_PARAMETER_CHANGE, so please can we do that? Sure. The log message

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
I can't see a reason why we would use a new WAL record for this, rather than modify the XLOG_PARAMETER_CHANGE record type which was created for a very similar reason. The code would be much simpler if we just extend XLOG_PARAMETER_CHANGE, so please can we do that? Sure. The log

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
Some testing notes -- select pg_start_backup('x'); ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
Some testing notes -- select pg_start_backup('x'); ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be using 'off' instead of 'invalid' since that is

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
Sorry. I was not previously able to answer fujii's all comments. This is the remaining answers. + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); + XLogCtl-Insert.fullPageWrites = fullPageWrites; + LWLockRelease(WALInsertLock); I don't think WALInsertLock needs to be hold here

Re: [HACKERS] Online base backup from the hot-standby

2011-10-13 Thread Jun Ishiduka
ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be using 'off' instead of 'invalid'

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Jun Ishiduka
As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record makes XLogReportParameters() quite confusing. Thought? I want to confirm the reply of Simon. I think we cannot decide how this

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Jun Ishiduka
if (!shutdown XLogStandbyInfoActive()) + { LogStandbySnapshot(checkPoint.oldestActiveXid, checkPoint.nextXid); + XLogReportParameters(REPORT_ON_BACKEND); + } Why doesn't the change of FPW need to be WAL-logged when shutdown checkpoint is

Re: [HACKERS] Online base backup from the hot-standby

2011-10-18 Thread Jun Ishiduka
+ /* + * The backend writes WAL of FPW at checkpoint. However, The backend do + * not need to write WAL of FPW at checkpoint shutdown because it + * performs when startup finishes. + */ +

Re: [HACKERS] Online base backup from the hot-standby

2011-10-18 Thread Jun Ishiduka
+ /* +* The backend writes WAL of FPW at checkpoint. However, The backend do +* not need to write WAL of FPW at checkpoint shutdown because it +* performs when startup finishes. +*/ +

Re: [HACKERS] Online base backup from the hot-standby

2011-10-19 Thread Jun Ishiduka
As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record makes XLogReportParameters() quite confusing. Thought? I updated a patch for what you have suggested (that the change of FPW