Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Fujii Masao
On Tue, Jan 24, 2012 at 7:54 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao masao.fu...@gmail.com wrote: I'll proceed to commit for this now. Thanks a lot! Can I just check a few things? Sure! You say /* +        * Update full_page_writes in

Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Simon Riggs
On Wed, Jan 25, 2012 at 8:16 AM, Fujii Masao masao.fu...@gmail.com wrote: What happens if we shutdown the WALwriter and then issue SIGHUP? SIGHUP doesn't affect full_page_writes in that case. Oh, you are concerned about the case where smart shutdown kills walwriter but some backends are

Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Simon Riggs
On Wed, Jan 25, 2012 at 8:49 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Jan 25, 2012 at 8:16 AM, Fujii Masao masao.fu...@gmail.com wrote: What happens if we shutdown the WALwriter and then issue SIGHUP? SIGHUP doesn't affect full_page_writes in that case. Oh, you are concerned

Re: [HACKERS] Online base backup from the hot-standby

2012-01-25 Thread Fujii Masao
On Thu, Jan 26, 2012 at 3:07 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Jan 25, 2012 at 8:49 AM, Simon Riggs si...@2ndquadrant.com wrote: On Wed, Jan 25, 2012 at 8:16 AM, Fujii Masao masao.fu...@gmail.com wrote: What happens if we shutdown the WALwriter and then issue SIGHUP?

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Fujii Masao
On Mon, Jan 23, 2012 at 10:11 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jan 23, 2012 at 10:29 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao masao.fu...@gmail.com

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao masao.fu...@gmail.com wrote: I'll proceed to commit for this now. Thanks a lot! Can I just check a few things? You say /* +* Update full_page_writes in shared memory and write an +* XLOG_FPW_CHANGE record before resource manager

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 10:54 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao masao.fu...@gmail.com wrote: I'll proceed to commit for this now. Thanks a lot! Can I just check a few things? Just to clarify, not expecting another patch version, just

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Fujii Masao
On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for the review! On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs si...@2ndquadrant.com wrote: I'm looking at this patch and wondering why

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Simon Riggs
On Mon, Jan 23, 2012 at 10:29 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for the review! On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Robert Haas
On Mon, Jan 23, 2012 at 5:29 AM, Fujii Masao masao.fu...@gmail.com wrote: If many people think the patch is not acceptable without such a safeguard, I will do that right now. That's my view. I think we ought to resolve this issue before commit, especially since it seems unclear that we know

Re: [HACKERS] Online base backup from the hot-standby

2012-01-23 Thread Robert Haas
On Mon, Jan 23, 2012 at 8:11 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 23, 2012 at 5:29 AM, Fujii Masao masao.fu...@gmail.com wrote: If many people think the patch is not acceptable without such a safeguard, I will do that right now. That's my view.  I think we ought to resolve

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Fujii Masao
On Fri, Jan 20, 2012 at 1:01 PM, Steve Singer ssinger...@sympatico.ca wrote: Here is my review of this verison of the patch. I think this patch has been in every CF for 9.2 and I feel it is getting close to being committed. Thanks for the review! Testing Review

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Erik Rijkers
On Fri, January 20, 2012 05:01, Steve Singer wrote: On 12-01-17 05:38 AM, Fujii Masao wrote: On Fri, Jan 13, 2012 at 5:02 PM, Fujii Masaomasao.fu...@gmail.com wrote: The amount of code changes to allow pg_basebackup to make a backup from the standby seems to be small. So I ended up merging

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Fujii Masao
On Fri, Jan 20, 2012 at 7:37 PM, Erik Rijkers e...@xs4all.nl wrote: I'm not sure, but it does look like this is the mystery bug that I encountered repeatedly already in 9.0devel; but I was never able to reproduce it reliably.  But I don't think it was ever solved.  

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Simon Riggs
On Tue, Jan 17, 2012 at 10:38 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 13, 2012 at 5:02 PM, Fujii Masao masao.fu...@gmail.com wrote: The amount of code changes to allow pg_basebackup to make a backup from the standby seems to be small. So I ended up merging that changes and the

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Simon Riggs
On Fri, Jan 20, 2012 at 11:04 AM, Fujii Masao masao.fu...@gmail.com wrote: But Steve encountered it again, which means that the above fix is not sufficient. Unless the issue is derived from my patch, we should do another cycle of diagnosis of it. It's my bug, and I've posted a fix but not yet

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Fujii Masao
Thanks for the review! On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs si...@2ndquadrant.com wrote: I'm looking at this patch and wondering why we're doing so many press-ups to ensure full_page_writes parameter is on. This will still fail if you use a utility that removes the full page writes,

Re: [HACKERS] Online base backup from the hot-standby

2012-01-20 Thread Simon Riggs
On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao masao.fu...@gmail.com wrote: Thanks for the review! On Fri, Jan 20, 2012 at 8:15 PM, Simon Riggs si...@2ndquadrant.com wrote: I'm looking at this patch and wondering why we're doing so many press-ups to ensure full_page_writes parameter is on.

Re: [HACKERS] Online base backup from the hot-standby

2012-01-19 Thread Steve Singer
On 12-01-17 05:38 AM, Fujii Masao wrote: On Fri, Jan 13, 2012 at 5:02 PM, Fujii Masaomasao.fu...@gmail.com wrote: The amount of code changes to allow pg_basebackup to make a backup from the standby seems to be small. So I ended up merging that changes and the infrastructure patch. WIP patch

Re: [HACKERS] Online base backup from the hot-standby

2011-11-14 Thread Steve Singer
On 11-10-31 12:11 AM, Jun Ishiduka wrote: Agreed. I'll extract FPW stuff from the patch that I submitted, and revise it as the infrastructure patch. The changes of pg_start_backup() etc that Ishiduka-san did are also a server-side infrastructure. I will extract them as another infrastructure

Re: [HACKERS] Online base backup from the hot-standby

2011-11-03 Thread Josh Berkus
On 10/25/11 5:03 AM, Magnus Hagander wrote: If we want something to go in early, that could be as simple as a version of pg_basebackup that runs against the slave but only if full_page_writes=on on the master. If it's not, it throws an error. Then we can improve upon that by adding handling of

Re: [HACKERS] Online base backup from the hot-standby

2011-11-03 Thread Fujii Masao
On Fri, Nov 4, 2011 at 8:06 AM, Josh Berkus j...@agliodbs.com wrote: On 10/25/11 5:03 AM, Magnus Hagander wrote: If we want something to go in early, that could be as simple as a version of pg_basebackup that runs against the slave but only if full_page_writes=on on the master. If it's not, it

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Heikki Linnakangas
On 25.10.2011 08:12, Fujii Masao wrote: On Tue, Oct 25, 2011 at 12:24 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 24.10.2011 15:29, Fujii Masao wrote: +listitem +para + Copy the pg_control file from the cluster directory to the global + sub-directory of the

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Fujii Masao
On Tue, Oct 25, 2011 at 3:44 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: +para +      Again connect to the database as a superuser, and execute +functionpg_stop_backup/. This terminates the backup mode, but does not +      perform a switch to the next WAL segment, create

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Magnus Hagander
On Tue, Oct 25, 2011 at 10:50, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Oct 25, 2011 at 3:44 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: +para +      Again connect to the database as a superuser, and execute +functionpg_stop_backup/. This terminates the backup

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Fujii Masao
On Tue, Oct 25, 2011 at 7:19 PM, Magnus Hagander mag...@hagander.net wrote: I don't think we should necessarily give up completely. But doing a pg_basebackup way *first* seems reasonable - because it's going to be the easiest one to get right, given that we have more control there. Doesn't

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Magnus Hagander
On Tue, Oct 25, 2011 at 13:54, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Oct 25, 2011 at 7:19 PM, Magnus Hagander mag...@hagander.net wrote: I don't think we should necessarily give up completely. But doing a pg_basebackup way *first* seems reasonable - because it's going to be the

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Steve Singer
On 11-10-25 02:44 AM, Heikki Linnakangas wrote: With pg_basebackup, we have a fighting chance of getting this right, because we have more control over how the backup is made. For example, we can co-operate with the buffer manager to avoid torn-pages, eliminating the need for

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Heikki Linnakangas
On 25.10.2011 15:56, Steve Singer wrote: On 11-10-25 02:44 AM, Heikki Linnakangas wrote: With pg_basebackup, we have a fighting chance of getting this right, because we have more control over how the backup is made. For example, we can co-operate with the buffer manager to avoid torn-pages,

Re: [HACKERS] Online base backup from the hot-standby

2011-10-25 Thread Fujii Masao
On Tue, Oct 25, 2011 at 9:03 PM, Magnus Hagander mag...@hagander.net wrote: On Tue, Oct 25, 2011 at 13:54, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Oct 25, 2011 at 7:19 PM, Magnus Hagander mag...@hagander.net wrote: I don't think we should necessarily give up completely. But doing a

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Heikki Linnakangas
On 24.10.2011 15:29, Fujii Masao wrote: +listitem + para + Copy the pg_control file from the cluster directory to the global + sub-directory of the backup. For example: + programlisting + cp $PGDATA/global/pg_control /mnt/server/backupdir/global + /programlisting + /para +

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Heikki Linnakangas
On 24.10.2011 15:29, Fujii Masao wrote: In your patch, FPW is always WAL-logged at startup even when FPW has not been changed since last shutdown. I don't think that's required. I changed the recovery code so that it keeps track of last FPW indicated by WAL record. Then, at end of startup, if

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Robert Haas
On Mon, Oct 24, 2011 at 11:33 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 24.10.2011 15:29, Fujii Masao wrote: In your patch, FPW is always WAL-logged at startup even when FPW has not been changed since last shutdown. I don't think that's required. I changed the

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Fujii Masao
Thanks for the review! On Tue, Oct 25, 2011 at 12:24 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 24.10.2011 15:29, Fujii Masao wrote: +    listitem +     para +      Copy the pg_control file from the cluster directory to the global +      sub-directory of the

Re: [HACKERS] Online base backup from the hot-standby

2011-10-24 Thread Fujii Masao
On Tue, Oct 25, 2011 at 12:33 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: One problem with this whole FPW-tracking is that pg_lesslog makes it fail. I'm not sure what we need to do about that - maybe just add a warning to the docs. But it leaves a bit bad feeling in my

Re: [HACKERS] Online base backup from the hot-standby

2011-10-19 Thread Jun Ishiduka
As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record makes XLogReportParameters() quite confusing. Thought? I updated a patch for what you have suggested (that the change of FPW

Re: [HACKERS] Online base backup from the hot-standby

2011-10-18 Thread Jun Ishiduka
+ /* + * The backend writes WAL of FPW at checkpoint. However, The backend do + * not need to write WAL of FPW at checkpoint shutdown because it + * performs when startup finishes. + */ +

Re: [HACKERS] Online base backup from the hot-standby

2011-10-18 Thread Jun Ishiduka
+ /* +* The backend writes WAL of FPW at checkpoint. However, The backend do +* not need to write WAL of FPW at checkpoint shutdown because it +* performs when startup finishes. +*/ +

Re: [HACKERS] Online base backup from the hot-standby

2011-10-17 Thread Fujii Masao
2011/10/15 Jun Ishiduka ishizuka@po.ntts.co.jp:     if (!shutdown XLogStandbyInfoActive()) +   {             LogStandbySnapshot(checkPoint.oldestActiveXid, checkPoint.nextXid); +           XLogReportParameters(REPORT_ON_BACKEND); +   } Why doesn't the change of FPW need to

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Fujii Masao
2011/10/13 Jun Ishiduka ishizuka@po.ntts.co.jp: I updated to patch corresponded above-comments. Thanks for updating the patch! As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Jun Ishiduka
As I suggested in the reply to Simon, I think that the change of FPW should be WAL-logged separately from that of HS parameters. ISTM packing them in one WAL record makes XLogReportParameters() quite confusing. Thought? I want to confirm the reply of Simon. I think we cannot decide how this

Re: [HACKERS] Online base backup from the hot-standby

2011-10-14 Thread Jun Ishiduka
if (!shutdown XLogStandbyInfoActive()) + { LogStandbySnapshot(checkPoint.oldestActiveXid, checkPoint.nextXid); + XLogReportParameters(REPORT_ON_BACKEND); + } Why doesn't the change of FPW need to be WAL-logged when shutdown checkpoint is

Re: [HACKERS] Online base backup from the hot-standby

2011-10-13 Thread Jun Ishiduka
ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be using 'off' instead of 'invalid'

Re: [HACKERS] Online base backup from the hot-standby

2011-10-13 Thread Fujii Masao
On Mon, Oct 10, 2011 at 3:56 AM, Simon Riggs si...@2ndquadrant.com wrote: 2011/10/9 Jun Ishiduka ishizuka@po.ntts.co.jp:  Insert WAL including a value of current FPW (on master)   * In the the same timing as update, they insert WAL (is named     XLOG_FPW_CHANGE). XLOG_FPW_CHANGE has a

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
Some testing notes -- select pg_start_backup('x'); ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Fujii Masao
2011/10/12 Jun Ishiduka ishizuka@po.ntts.co.jp: ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest checkpoint We should be using 'off' instead of 'invalid' since that is

Re: [HACKERS] Online base backup from the hot-standby

2011-10-12 Thread Jun Ishiduka
Sorry. I was not previously able to answer fujii's all comments. This is the remaining answers. + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); + XLogCtl-Insert.fullPageWrites = fullPageWrites; + LWLockRelease(WALInsertLock); I don't think WALInsertLock needs to be hold here

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
I can't see a reason why we would use a new WAL record for this, rather than modify the XLOG_PARAMETER_CHANGE record type which was created for a very similar reason. The code would be much simpler if we just extend XLOG_PARAMETER_CHANGE, so please can we do that? Sure. The log message

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
I can't see a reason why we would use a new WAL record for this, rather than modify the XLOG_PARAMETER_CHANGE record type which was created for a very similar reason. The code would be much simpler if we just extend XLOG_PARAMETER_CHANGE, so please can we do that? Sure. The log

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Steve Singer
On 11-10-11 11:17 AM, Jun Ishiduka wrote: Done. Updated patch attached. I have taken Jun's latest patch and applied it on top of Fujii's most recent patch. I did some testing with the result but nothing theory enough to stumble on any race conditions. Some testing notes

Re: [HACKERS] Online base backup from the hot-standby

2011-10-11 Thread Jun Ishiduka
Some testing notes -- select pg_start_backup('x'); ERROR: full_page_writes on master is set invalid at least once since latest checkpoint I think this error should be rewritten as ERROR: full_page_writes on master has been off at some point since latest

Re: [HACKERS] Online base backup from the hot-standby

2011-10-09 Thread Jun Ishiduka
I created a patch corresponding FPW. Fujii's patch (ver 9) is based. Manage own FPW in shared-memory (on master) * startup and walwriter process update it. startup initializes it after REDO. walwriter updates it when started or received SIGHUP. Insert WAL including a value of current

Re: [HACKERS] Online base backup from the hot-standby

2011-10-09 Thread Simon Riggs
2011/10/9 Jun Ishiduka ishizuka@po.ntts.co.jp:  Insert WAL including a value of current FPW (on master)   * In the the same timing as update, they insert WAL (is named     XLOG_FPW_CHANGE). XLOG_FPW_CHANGE has a value of the changed FPW.   * When it creates CHECKPOINT, it adds a value of

Re: [HACKERS] Online base backup from the hot-standby

2011-09-27 Thread Steve Singer
On 11-09-26 10:56 PM, Fujii Masao wrote: Looks weired. Though the WAL record starting from 0/6000298 was read successfully, then re-fetch of the same record fails at the end of recovery. One possible cause is the corruption of archived WAL file. What restore_command on the standby and

Re: [HACKERS] Online base backup from the hot-standby

2011-09-27 Thread Fujii Masao
On Wed, Sep 28, 2011 at 8:10 AM, Steve Singer ssinger...@sympatico.ca wrote: This is the test procedure I'm trying today, I wasn't able to reproduce the crash.  What I was doing the other day was similar but I can't speak to unintentional differences. Thanks for the info! I tried your test

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Jun Ishiduka
Attached is the updated version of the patch. I refactored the code, fixed some bugs, added lots of source code comments, improved the document, but didn't change the basic design. Please check this patch, and let's use this patch as the base if you agree with that. Thanks for update patch.

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Fujii Masao
On Fri, Sep 23, 2011 at 12:44 AM, Magnus Hagander mag...@hagander.net wrote: Would it make sense for pg_start_backup() to have the ability to wait for the next restartpoint in a case like this, if we know that FPW has been set? Instead of failing? Or maybe that's just overcomplicating things

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Fujii Masao
On Mon, Sep 26, 2011 at 11:39 AM, Steve Singer ssinger...@sympatico.ca wrote: I have looked at both Jun's patch from Sept 13 and Fujii's updates to the patch.  I agree that Fujii's updated version should be used as the basis for changes going forward.   My comments below refer to that version

Re: [HACKERS] Online base backup from the hot-standby

2011-09-26 Thread Fujii Masao
On Tue, Sep 27, 2011 at 11:56 AM, Fujii Masao masao.fu...@gmail.com wrote: In backup.sgml  the new section titled Making a Base Backup during Recovery  I would prefer to see some mention in the title that this procedure is for standby servers ie Making a Base Backup from a Standby Database. 

Re: [HACKERS] Online base backup from the hot-standby

2011-09-25 Thread Steve Singer
On 11-09-22 09:24 AM, Fujii Masao wrote: On Wed, Sep 21, 2011 at 11:50 AM, Fujii Masaomasao.fu...@gmail.com wrote: 2011/9/13 Jun Ishidukaishizuka@po.ntts.co.jp: Update patch. Changes: * set 'on' full_page_writes by user (in document) * read FROM: XX in backup_label (in xlog.c) *

Re: [HACKERS] Online base backup from the hot-standby

2011-09-22 Thread Fujii Masao
On Wed, Sep 21, 2011 at 5:34 PM, Magnus Hagander mag...@hagander.net wrote: On Wed, Sep 21, 2011 at 08:23, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander mag...@hagander.net wrote: Presumably pg_start_backup() will check this. And we'll somehow track

Re: [HACKERS] Online base backup from the hot-standby

2011-09-22 Thread Fujii Masao
On Wed, Sep 21, 2011 at 11:50 AM, Fujii Masao masao.fu...@gmail.com wrote: 2011/9/13 Jun Ishiduka ishizuka@po.ntts.co.jp: Update patch. Changes:  * set 'on' full_page_writes by user (in document)  * read FROM: XX in backup_label (in xlog.c)  * check status when pg_stop_backup is

Re: [HACKERS] Online base backup from the hot-standby

2011-09-22 Thread Magnus Hagander
On Thu, Sep 22, 2011 at 14:13, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 21, 2011 at 5:34 PM, Magnus Hagander mag...@hagander.net wrote: On Wed, Sep 21, 2011 at 08:23, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander mag...@hagander.net

Re: [HACKERS] Online base backup from the hot-standby

2011-09-21 Thread Fujii Masao
On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander mag...@hagander.net wrote: On Wed, Sep 21, 2011 at 04:50, Fujii Masao masao.fu...@gmail.com wrote: 3. Copy the pg_control file from the cluster directory on the standby to    the backup as follows:    cp $PGDATA/global/pg_control

Re: [HACKERS] Online base backup from the hot-standby

2011-09-21 Thread Magnus Hagander
On Wed, Sep 21, 2011 at 08:23, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Sep 21, 2011 at 2:13 PM, Magnus Hagander mag...@hagander.net wrote: On Wed, Sep 21, 2011 at 04:50, Fujii Masao masao.fu...@gmail.com wrote: 3. Copy the pg_control file from the cluster directory on the standby to  

Re: [HACKERS] Online base backup from the hot-standby

2011-09-20 Thread Fujii Masao
2011/9/13 Jun Ishiduka ishizuka@po.ntts.co.jp: Update patch. Changes:  * set 'on' full_page_writes by user (in document)  * read FROM: XX in backup_label (in xlog.c)  * check status when pg_stop_backup is executed (in xlog.c) Thanks for updating the patch. Before reviewing the patch,

Re: [HACKERS] Online base backup from the hot-standby

2011-09-20 Thread Magnus Hagander
On Wed, Sep 21, 2011 at 04:50, Fujii Masao masao.fu...@gmail.com wrote: 2011/9/13 Jun Ishiduka ishizuka@po.ntts.co.jp: Update patch. Changes:  * set 'on' full_page_writes by user (in document)  * read FROM: XX in backup_label (in xlog.c)  * check status when pg_stop_backup is executed

Re: [HACKERS] Online base backup from the hot-standby

2011-09-13 Thread Jun Ishiduka
Update patch. Changes: * set 'on' full_page_writes by user (in document) * read FROM: XX in backup_label (in xlog.c) * check status when pg_stop_backup is executed (in xlog.c) Hi, Created a patch in response to comments. * Procedure 1. Call pg_start_backup('x') on hot standby. 2.

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Robert Haas
On Wed, Aug 17, 2011 at 6:19 AM, Fujii Masao masao.fu...@gmail.com wrote: 2011/8/17 Jun Ishiduka ishizuka@po.ntts.co.jp: I see in xlog.h XLR_BKP_REMOVABLE, the comment above it says that this flag is used to indicate that the archiver can compress the full page blocks to non-full page

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
On Wed, Aug 17, 2011 at 9:40 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 17, 2011 at 6:19 AM, Fujii Masao masao.fu...@gmail.com wrote: The straightforward approach to address the problem you raised is to log the change of full_page_writes on the master. Since such a WAL record is

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Robert Haas
On Wed, Aug 17, 2011 at 9:53 AM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Aug 17, 2011 at 9:40 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Aug 17, 2011 at 6:19 AM, Fujii Masao masao.fu...@gmail.com wrote: The straightforward approach to address the problem you raised is to log

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
On Thu, Aug 18, 2011 at 12:09 AM, Robert Haas robertmh...@gmail.com wrote: Ugh, you're right.  But then you might have problems if the state changes again before all backends have picked up the previous change. Right. What I've thought about before is making one backend (say, bgwriter) store

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Fujii Masao
2011/8/5 Jun Ishiduka ishizuka@po.ntts.co.jp: * Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of the data dir. 3. Call pg_stop_backup() on the standby. 4. Copy the control file on the standby to the backup. 5. Check whether the control file is status during hot

Re: [HACKERS] Online base backup from the hot-standby

2011-08-17 Thread Jun Ishiduka
* Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of the data dir. 3. Call pg_stop_backup() on the standby. 4. Copy the control file on the standby to the backup. 5. Check whether the control file is status during hot standby with pg_controldata. ? - If

Re: [HACKERS] Online base backup from the hot-standby

2011-08-16 Thread Jun Ishiduka
* Not correspond yet ?* full_page_write = off ? ?- If the primary is full_page_write = off, archive recovery may not act ? ? ? normally. Therefore the standby may need to check whether full_page_write ? ? ? = off to WAL. Isn't having a standby make the

Re: [HACKERS] Online base backup from the hot-standby

2011-08-16 Thread Jun Ishiduka
* Not correspond yet ?* full_page_write = off ? ?- If the primary is full_page_write = off, archive recovery may not act ? ? ? normally. Therefore the standby may need to check whether full_page_write ? ? ? = off to WAL. Isn't having a standby make the

Re: [HACKERS] Online base backup from the hot-standby

2011-08-16 Thread Steve Singer
On 11-08-16 02:09 AM, Jun Ishiduka wrote: Thanks. This has the following two problems. * pg_start_backup() must set 'on' to full_page_writes of the master that is actual writing of the WAL, but not the standby. Is there any way to tell from the WAL segments if they contain the full

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Jun Ishiduka
* Not correspond yet * full_page_write = off - If the primary is full_page_write = off, archive recovery may not act normally. Therefore the standby may need to check whether full_page_write = off to WAL. Isn't having a standby make the full_page_write = on in

Re: [HACKERS] Online base backup from the hot-standby

2011-08-15 Thread Robert Haas
2011/8/15 Jun Ishiduka ishizuka@po.ntts.co.jp: * Not correspond yet  * full_page_write = off    - If the primary is full_page_write = off, archive recovery may not act       normally. Therefore the standby may need to check whether full_page_write       = off to WAL. Isn't

[HACKERS] Online base backup from the hot-standby

2011-08-05 Thread Jun Ishiduka
I will provide a patch which can exeute pg_start/stop_backup including to solve above comment and conditions in next stage. Then please review. done. * Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of the data dir. 3. Call pg_stop_backup() on the standby. 4. Copy

Re: [HACKERS] Online base backup from the hot-standby

2011-08-05 Thread Cédric Villemain
2011/8/5 Jun Ishiduka ishizuka@po.ntts.co.jp: I will provide a patch which can exeute pg_start/stop_backup including to solve above comment and conditions in next stage. Then please review. done. great ! * Procedure 1. Call pg_start_backup('x') on the standby. 2. Take a backup of

Re: [HACKERS] Online base backup from the hot-standby

2011-07-12 Thread Jun Ishiduka
This version of the patch adds a field into pg_controldata that tries to store the source of the base backup while in recovery mode. I think your ultimate goal with this patch is to be able to take a backup of a running hot-standby slave and recover it as another instance. This patch seems

Re: [HACKERS] Online base backup from the hot-standby

2011-07-10 Thread Steve Singer
On 11-07-07 09:22 PM, Jun Ishiduka wrote: As you proposed, adding new field which stores the backup end location taken from minRecoveryPoint, into pg_control sounds good idea. Update patch. Here is a review of the updated patch This version of the patch adds a field into pg_controldata that

Re: [HACKERS] Online base backup from the hot-standby

2011-07-07 Thread Jun Ishiduka
As you proposed, adding new field which stores the backup end location taken from minRecoveryPoint, into pg_control sounds good idea. Update patch. Regards. Jun Ishizuka NTT Software Corporation TEL:045-317-7018 E-Mail: ishizuka@po.ntts.co.jp

Re: [HACKERS] Online base backup from the hot-standby

2011-07-05 Thread Jun Ishiduka
What about using backupStartPoint to check whether this recovery started from the backup or not? No, postgres can check whether this recovery started from the backup or not, but can not check whether standby server or master (got backup from). Once recovery started, backupStartPoint is

Re: [HACKERS] Online base backup from the hot-standby

2011-07-05 Thread Fujii Masao
2011/7/5 Jun Ishiduka ishizuka@po.ntts.co.jp: What about using backupStartPoint to check whether this recovery started from the backup or not? No, postgres can check whether this recovery started from the backup or not, but can not check whether standby server or master (got backup

Re: [HACKERS] Online base backup from the hot-standby

2011-07-04 Thread Jun Ishiduka
When the standby restarts after it crashes during recovery, it always checks whether recovery has reached the backup end location by using minRecoveryPoint even though the standby doesn't start from the backup. This looks odd. Certainly. But, in this case, the state before recovery starts

Re: [HACKERS] Online base backup from the hot-standby

2011-07-04 Thread Fujii Masao
2011/7/4 Jun Ishiduka ishizuka@po.ntts.co.jp: When the standby restarts after it crashes during recovery, it always checks whether recovery has reached the backup end location by using minRecoveryPoint even though the standby doesn't start from the backup. This looks odd. Certainly.

Re: [HACKERS] Online base backup from the hot-standby

2011-07-03 Thread Fujii Masao
2011/7/1 Jun Ishiduka ishizuka@po.ntts.co.jp: On this commitfest, the goal of the patch is to be able to be recovered using Minimum recovery ending location in the control file. Done. When the standby restarts after it crashes during recovery, it always checks whether recovery has

Re: [HACKERS] Online base backup from the hot-standby

2011-06-30 Thread Jun Ishiduka
Process of online base backup on standby server: 1. pg_start_backup('x'); 2. copy the data directory 3. copy *pg_control* Who deletes the backup_label file created by pg_start_backup()? Isn't pg_stop_backup() required to do that? You need it to take the system out of

Re: [HACKERS] Online base backup from the hot-standby

2011-06-30 Thread Jun Ishiduka
On this commitfest, the goal of the patch is to be able to be recovered using Minimum recovery ending location in the control file. Done. Regards. Jun Ishizuka NTT Software Corporation TEL:045-317-7018 E-Mail: ishizuka@po.ntts.co.jp

Re: [HACKERS] Online base backup from the hot-standby

2011-06-29 Thread Fujii Masao
2011/6/28 Jun Ishiduka ishizuka@po.ntts.co.jp: Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to accomplish base backups from standby servers? If not what changes do you think should be made? I reconsider the way to

Re: [HACKERS] Online base backup from the hot-standby

2011-06-29 Thread Magnus Hagander
On Jun 30, 2011 5:59 AM, Fujii Masao masao.fu...@gmail.com wrote: 2011/6/28 Jun Ishiduka ishizuka@po.ntts.co.jp: Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to accomplish base backups from standby servers? If

Re: [HACKERS] Online base backup from the hot-standby

2011-06-28 Thread Steve Singer
On 11-06-28 01:52 AM, Jun Ishiduka wrote: Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to accomplish base backups from standby servers? If not what changes do you think should be made? I reconsider the way to not use

Re: [HACKERS] Online base backup from the hot-standby

2011-06-27 Thread Jun Ishiduka
Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to accomplish base backups from standby servers? If not what changes do you think should be made? I reconsider the way to not use pg_stop_backup(). Process of online base

Re: [HACKERS] Online base backup from the hot-standby

2011-06-24 Thread Steve Singer
On 11-06-24 12:41 AM, Jun Ishiduka wrote: The logic that not use pg_stop_backup() would be difficult, because pg_stop_backup() is used to identify minRecoveryPoint. Considering everything that has been discussed on this thread so far. Do you still think your patch is the best way to

Re: [HACKERS] Online base backup from the hot-standby

2011-06-23 Thread Jun Ishiduka
1) Today you can do a backup by just calling pg_start_backup('x'); copy the data directory and pg_stop_backup(); You do not need to use pg_basebackup to create a backup. The solution you are proposing would require pg_basebackup to be used to build backups from standby servers. YES. 2)

Re: [HACKERS] Online base backup from the hot-standby

2011-06-23 Thread Steve Singer
On 11-06-23 02:41 AM, Jun Ishiduka wrote: I receive this mail, so I notice I do wrong recognition to what Heikki is proposing. my recognition: Before: * I thought Heikki proposes, Execute SQL(pg_start_backup('x'); copy the data directory and pg_stop_backup();) from the standby

Re: [HACKERS] Online base backup from the hot-standby

2011-06-23 Thread Jun Ishiduka
What I think he is proposing would not require using pg_stop_backup() but you could also modify pg_stop_back() to work as well. What do you think of this idea? Do you see how the patch can be reworked to accomplish this? The logic that not use pg_stop_backup() would be difficult, because

  1   2   >