Re: [HACKERS] Online base backup from the hot-standby
> As I suggested in the reply to Simon, I think that the change of FPW > should be WAL-logged separately from that of HS parameters. ISTM > packing them in one WAL record makes XLogReportParameters() > quite confusing. Thought? I updated a patch for what you have suggested (that the change of FPW should be WAL-logged separately from that of HS parameters). I want to base on this patch if there are no other opinions. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base-07fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > + /* > > +* The backend writes WAL of FPW at checkpoint. However, The > > backend do > > +* not need to write WAL of FPW at checkpoint shutdown because > > it > > +* performs when startup finishes. > > +*/ > > + XLogReportParameters(REPORT_ON_BACKEND); > > > > I'm still unclear why that WAL doesn't need to be written at shutdown > > checkpoint. > > Anyway, the first sentence in the above comments is not right. Not a > > backend but > > a bgwriter writes that WAL at checkpoint. > > > > The second also seems not to be right. It implies that a shutdown > > checkpoint is > > performed only at end of startup. But it may be done when smart or fast > > shutdown > > is requested. > > > Okay. > I change to the following messages. > > /* > * The bgwriter writes WAL of FPW at checkpoint. But does not at shutdown. > * Because XLogReportParameters() is always called at the end of startup > * process, it does not need to be called at shutdown. > */ > > > In addition, I change macro name. > > REPORT_ON_BACKEND -> REPORT_ON_BGWRITER I have updated as above-comment. Please check this. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base-06fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> + /* > + * The backend writes WAL of FPW at checkpoint. However, The > backend do > + * not need to write WAL of FPW at checkpoint shutdown because > it > + * performs when startup finishes. > + */ > + XLogReportParameters(REPORT_ON_BACKEND); > > I'm still unclear why that WAL doesn't need to be written at shutdown > checkpoint. > Anyway, the first sentence in the above comments is not right. Not a backend > but > a bgwriter writes that WAL at checkpoint. > > The second also seems not to be right. It implies that a shutdown checkpoint > is > performed only at end of startup. But it may be done when smart or fast > shutdown > is requested. Okay. I change to the following messages. /* * The bgwriter writes WAL of FPW at checkpoint. But does not at shutdown. * Because XLogReportParameters() is always called at the end of startup * process, it does not need to be called at shutdown. */ In addition, I change macro name. REPORT_ON_BACKEND -> REPORT_ON_BGWRITER Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > if (!shutdown && XLogStandbyInfoActive())
> > + {
> > LogStandbySnapshot(&checkPoint.oldestActiveXid,
> > &checkPoint.nextXid);
> > + XLogReportParameters(REPORT_ON_BACKEND);
> > + }
> >
> > Why doesn't the change of FPW need to be WAL-logged when
> > shutdown checkpoint is performed? It's helpful to add the comment
> > explaining why.
>
> Sure. I update the patch soon.
Done.
Please check this.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_09base-05fpw.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> As I suggested in the reply to Simon, I think that the change of FPW
> should be WAL-logged separately from that of HS parameters. ISTM
> packing them in one WAL record makes XLogReportParameters()
> quite confusing. Thought?
I want to confirm the reply of Simon. I think we cannot decide how this
code should be if there is not the reply.
> if (!shutdown && XLogStandbyInfoActive())
> + {
> LogStandbySnapshot(&checkPoint.oldestActiveXid,
> &checkPoint.nextXid);
> + XLogReportParameters(REPORT_ON_BACKEND);
> + }
>
> Why doesn't the change of FPW need to be WAL-logged when
> shutdown checkpoint is performed? It's helpful to add the comment
> explaining why.
Sure. I update the patch soon.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > > > > ERROR: full_page_writes on master is set invalid at least once since > > > > latest checkpoint > > > > > > > > I think this error should be rewritten as > > > > ERROR: full_page_writes on master has been off at some point since > > > > latest checkpoint > > > > > > > > We should be using 'off' instead of 'invalid' since that is what is what > > > > the user sets it to. > > > > > > Sure. > > > > What about the following message? It sounds more precise to me. > > > > ERROR: WAL generated with full_page_writes=off was replayed since last > > restartpoint > > Okay, I changes the patch to this messages. > If someone says there is a idea better than it, I will consider again. > > > > > I updated to patch corresponded above-comments. > > > > Thanks for updating the patch! Here are the comments: > > > > * don't yet have the insert lock, forcePageWrites could change under > > us, > > * but we'll recheck it once we have the lock. > > */ > > - doPageWrites = fullPageWrites || Insert->forcePageWrites; > > + doPageWrites = Insert->fullPageWrites || Insert->forcePageWrites; > > > > The source comment needs to be modified. > > > > * just turned off, we could recompute the record without full pages, > > but > > * we choose not to bother.) > > */ > > - if (Insert->forcePageWrites && !doPageWrites) > > + if ((Insert->fullPageWrites || Insert->forcePageWrites) && > > !doPageWrites) > > > > Same as above. > > Sure. > > > > XLogReportParameters() should skip writing WAL if full_page_writes has not > > been > > changed by SIGHUP. > > > > XLogReportParameters() should skip updating pg_control if any parameter > > related > > to hot standby has not been changed. > > YES. > > > > In checkpoint, XLogReportParameters() is called only when wal_level is > > hot_standby. > > OTOH, in walwriter, it's always called even when wal_level is not > > hot_standby. > > Can't we skip calling XLogReportParameters() whenever wal_level is not > > hot_standby? > > Yes, It is possible. > > > > In do_pg_start_backup() and do_pg_stop_backup(), the spinlock must be held > > to > > see XLogCtl->lastFpwDisabledLSN. > > Yes. > > > > What about changing the error message to: > > ERROR: WAL generated with full_page_writes=off was replayed during online > > backup > > Okay, too. > Sorry. > I was not previously able to answer fujii's all comments. > This is the remaining answers. > > > > + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); > > + XLogCtl->Insert.fullPageWrites = fullPageWrites; > > + LWLockRelease(WALInsertLock); > > > > I don't think WALInsertLock needs to be hold here because there is no > > concurrently running process which can access Insert.fullPageWrites. > > For example, Insert->currpos and Insert->LogwrtResult are also changed > > without the lock there. > > > > Yes. > > > The source comment of XLogReportParameters() needs to be modified. > > Yes, too. Done. I updated to patch corresponded above-comments. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base-04fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Sorry. I was not previously able to answer fujii's all comments. This is the remaining answers. > + LWLockAcquire(WALInsertLock, LW_EXCLUSIVE); > + XLogCtl->Insert.fullPageWrites = fullPageWrites; > + LWLockRelease(WALInsertLock); > > I don't think WALInsertLock needs to be hold here because there is no > concurrently running process which can access Insert.fullPageWrites. > For example, Insert->currpos and Insert->LogwrtResult are also changed > without the lock there. > Yes. > The source comment of XLogReportParameters() needs to be modified. Yes, too. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > > ERROR: full_page_writes on master is set invalid at least once since > > > latest checkpoint > > > > > > I think this error should be rewritten as > > > ERROR: full_page_writes on master has been off at some point since > > > latest checkpoint > > > > > > We should be using 'off' instead of 'invalid' since that is what is what > > > the user sets it to. > > > > Sure. > > What about the following message? It sounds more precise to me. > > ERROR: WAL generated with full_page_writes=off was replayed since last > restartpoint Okay, I changes the patch to this messages. If someone says there is a idea better than it, I will consider again. > > I updated to patch corresponded above-comments. > > Thanks for updating the patch! Here are the comments: > >* don't yet have the insert lock, forcePageWrites could change under > us, >* but we'll recheck it once we have the lock. >*/ > - doPageWrites = fullPageWrites || Insert->forcePageWrites; > + doPageWrites = Insert->fullPageWrites || Insert->forcePageWrites; > > The source comment needs to be modified. > >* just turned off, we could recompute the record without full pages, > but >* we choose not to bother.) >*/ > - if (Insert->forcePageWrites && !doPageWrites) > + if ((Insert->fullPageWrites || Insert->forcePageWrites) && > !doPageWrites) > > Same as above. Sure. > XLogReportParameters() should skip writing WAL if full_page_writes has not > been > changed by SIGHUP. > > XLogReportParameters() should skip updating pg_control if any parameter > related > to hot standby has not been changed. YES. > In checkpoint, XLogReportParameters() is called only when wal_level is > hot_standby. > OTOH, in walwriter, it's always called even when wal_level is not hot_standby. > Can't we skip calling XLogReportParameters() whenever wal_level is not > hot_standby? Yes, It is possible. > In do_pg_start_backup() and do_pg_stop_backup(), the spinlock must be held to > see XLogCtl->lastFpwDisabledLSN. Yes. > What about changing the error message to: > ERROR: WAL generated with full_page_writes=off was replayed during online > backup Okay, too. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > Some testing notes
> > --
> > select pg_start_backup('x');
> > ERROR: full_page_writes on master is set invalid at least once since
> > latest checkpoint
> >
> > I think this error should be rewritten as
> > ERROR: full_page_writes on master has been off at some point since
> > latest checkpoint
> >
> > We should be using 'off' instead of 'invalid' since that is what is what
> > the user sets it to.
>
> Sure.
> > Minor typo above at 'CHECKPOTNT'
>
> Yes.
I updated to patch corresponded above-comments.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_09base-03fpw.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> Some testing notes
> --
> select pg_start_backup('x');
> ERROR: full_page_writes on master is set invalid at least once since
> latest checkpoint
>
> I think this error should be rewritten as
> ERROR: full_page_writes on master has been off at some point since
> latest checkpoint
>
> We should be using 'off' instead of 'invalid' since that is what is what
> the user sets it to.
Sure.
> I switched full_page_writes=on , on the master
>
> did a pg_start_backup() on the slave1.
>
> Then I switched full_page_writes=off on the master, did a reload +
> checkpoint.
>
> I was able to then do my backup of slave1, copy the control file, and
> pg_stop_backup().
>
> When I did the test slave2 started okay, but is this safe? Do we need a
> warning from pg_stop_backup() that is printed if it is detected that
> full_page_writes was turned off on the master during the backup period?
I also reproduced.
pg_stop_backup() fails in most cases.
However, it succeeds if both the following cases are true.
* checkpoint is done before walwriter recieves SIGHUP.
* slave1 has not received the WAL of 'off' by SIGHUP yet.
> Minor typo above at 'CHECKPOTNT'
Yes.
> If my concern about full page writes being switched to off in the middle
> of a backup is unfounded then I think this patch is ready for a
> committer. They can clean the two editorial changes when they apply the
> patches.
Yes. I'll clean since these comments fix.
> If do_pg_stop_backup is going to need some logic to recheck the full
> page write status then an updated patch is required.
It already contains.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > I can't see a reason why we would use a new WAL record for this,
> > rather than modify the XLOG_PARAMETER_CHANGE record type which was
> > created for a very similar reason.
> > The code would be much simpler if we just extend
> > XLOG_PARAMETER_CHANGE, so please can we do that?
>
> Sure.
>
> > The log message "full_page_writes on master is set invalid more than
> > once during online backup" should read "at least once" rather than
> > "more than once".
>
> Yes.
>
> > lastFpwDisabledLSN needs to be initialized.
>
> I think it don't need because all values in XLogCtl is initialized 0.
>
> > Is there a reason to add lastFpwDisabledLSN onto the Control file? If
> > we log parameters after every checkpoint then we'll know the values
> > when we startup. If we keep logging parameters this way we'll end up
> > with a very awkward and large control file. I would personally prefer
> > to avoid that, but that thought could go either way. Let's see if
> > anyone else thinks that also.
>
> Yes. I add to CreateCheckPoint().
>
> Image:
> CreateCheckPoint()
> {
> if (!shutdown && XLogStandbyInfoActive())
> {
> LogStandbySnapshot()
> XLogReportParameters()
> }
>}
>
> XLogReportParameters()
> {
> if (fpw == 'off' || ... )
> XLOGINSERT()
> }
>
> However, it'll write XLOG_PARAMETER_CHANGE every checkpoints when FPW is
> 'off'.
> (It will increases the amount of WAL.)
> Is it OK?
Done.
Updated patch attached.
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_09base-02fpw.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> I can't see a reason why we would use a new WAL record for this,
> rather than modify the XLOG_PARAMETER_CHANGE record type which was
> created for a very similar reason.
> The code would be much simpler if we just extend
> XLOG_PARAMETER_CHANGE, so please can we do that?
Sure.
> The log message "full_page_writes on master is set invalid more than
> once during online backup" should read "at least once" rather than
> "more than once".
Yes.
> lastFpwDisabledLSN needs to be initialized.
I think it don't need because all values in XLogCtl is initialized 0.
> Is there a reason to add lastFpwDisabledLSN onto the Control file? If
> we log parameters after every checkpoint then we'll know the values
> when we startup. If we keep logging parameters this way we'll end up
> with a very awkward and large control file. I would personally prefer
> to avoid that, but that thought could go either way. Let's see if
> anyone else thinks that also.
Yes. I add to CreateCheckPoint().
Image:
CreateCheckPoint()
{
if (!shutdown && XLogStandbyInfoActive())
{
LogStandbySnapshot()
XLogReportParameters()
}
}
XLogReportParameters()
{
if (fpw == 'off' || ... )
XLOGINSERT()
}
However, it'll write XLOG_PARAMETER_CHANGE every checkpoints when FPW is 'off'.
(It will increases the amount of WAL.)
Is it OK?
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
I created a patch corresponding FPW. Fujii's patch (ver 9) is based. Manage own FPW in shared-memory (on master) * startup and walwriter process update it. startup initializes it after REDO. walwriter updates it when started or received SIGHUP. Insert WAL including a value of current FPW (on master) * In the the same timing as update, they insert WAL (is named XLOG_FPW_CHANGE). XLOG_FPW_CHANGE has a value of the changed FPW. * When it creates CHECKPOINT, it adds a value of current FPW to the CHECKPOINT WAL. Manage master's FPW in local-memory in startup (on standby) * It takes a value of the master's FPW by reading XLOG_FPW_CHANGE at REDO. Check when pg_start_backup/pg_stop_backup (on standby) * It checks to use these two value. * master's FPW at latest CHECKPOINT * current master's FPW by XLOG_FPW_CHANGE Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_09base_01fpw.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> Attached is the updated version of the patch. I refactored the code, fixed > some bugs, added lots of source code comments, improved the document, > but didn't change the basic design. Please check this patch, and let's use > this patch as the base if you agree with that. Thanks for update patch. Yes. I agree. > In the current patch, there is no safeguard for preventing users from > taking backup during recovery when FPW is disabled. This is unsafe. > Are you planning to implement such a safeguard? Yes. I want to reference the following Fujii's comments. - > Right. Let me explain again what I'm thinking. > > When FPW is changed, the master always writes the WAL record > which contains the current value of FPW. This means that the standby > can track all changes of FPW by reading WAL records. > > The standby has two flags: One indicates whether FPW has always > been TRUE since last restartpoint. Another indicates whether FPW > has always been TRUE since last pg_start_backup(). The standby > can maintain those flags by reading WAL records streamed from > the master. > > If the former flag indicates FALSE (i.e., the WAL records which > the standby has replayed since last restartpoint might not contain > required FPW), pg_start_backup() fails. If the latter flag indicates > FALSE (i.e., the WAL records which the standby has replayed > during the backup might not contain required FPW), > pg_stop_backup() fails. > > If I'm not missing something, this approach can address the problem > which you're concerned about. - Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
Update patch.
Changes:
* set 'on' full_page_writes by user (in document)
* read "FROM: XX" in backup_label (in xlog.c)
* check status when pg_stop_backup is executed (in xlog.c)
> Hi, Created a patch in response to comments.
>
>
> * Procedure
> 1. Call pg_start_backup('x') on hot standby.
> 2. Take a backup of the data dir.
> 3. Copy the control file on hot standby to the backup.
> 4. Call pg_stop_backup() on hot standby.
>
>
> * Behavior
> (take backup)
> If we execute pg_start_backup() on hot standby then execute restartpoint,
> write a strings as "FROM: slave" in backup_label and change backup mode,
> but do not change full_page_writes into "on" forcibly.
>
> If we execute pg_stop_backup() on hot standby then rename backup_label
> and change backup mode, but neither write backup end record and history
> file nor wait to complete the WAL archiving.
> pg_stop_backup() is returned this MinRecoveryPoint as result.
>
> If we execute pg_stop_backup() on the server promoted then error
> message is output since read the backup_label.
>
> (recovery)
> If we recover with the backup taken on hot standby, MinRecoveryPoint in
> the control file copied by 3 of above-procedure is used instead of backup
> end record.
>
> If recovery starts as first, BackupEndPoint in the control file is written
> a same value as MinRecoveryPoint. This is for remembering the value of
> MinRecoveryPoint during recovery.
>
> HINT message("If this has ...") is always output when we recover with the
> backup taken on hot standby.
>
>
> * Problem
> full_page_writes's problem.
> > This has the following two problems.
> > * pg_start_backup() must set 'on' to full_page_writes of the master that
> >is actual writing of the WAL, but not the standby.
> > * The standby doesn't need to connect to the master that's actual
> writing
> >WAL.
> >(Ex. Standby2 in Cascade Replication: Master - Standby1 - Standby2)
> >
> > I'm worried how I should clear these problems.
>
> Status: Considering
> (Latest: http://archives.postgresql.org/pgsql-hackers/2011-08/msg00880.php)
>
>
> Regards.
>
>
>
> Jun Ishizuka
> NTT Software Corporation
> TEL：045-317-7018
> E-Mail: ishizuka@po.ntts.co.jp
>
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_07.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > * Procedure
> >
> > 1. Call pg_start_backup('x') on the standby.
> > 2. Take a backup of the data dir.
> > 3. Call pg_stop_backup() on the standby.
> > 4. Copy the control file on the standby to the backup.
> > 5. Check whether the control file is status during hot standby with
> > pg_controldata.
> > ? -> If the standby promote between 3. and 4., the backup can not recovery.
> > ? ? ?-> pg_control is that "Minimum recovery ending location" is equals 0/0.
> > ? ? ?-> backup-end record is not written.
>
> What if we do #4 before #3? The backup gets corrupted? My guess is
> that the backup is still valid even if we copy pg_control before executing
> pg_stop_backup(). Which would not require #5 because if the standby
> promotion happens before pg_stop_backup(), pg_stop_backup() can
> detect that status change and cancel the backup.
>
> #5 looks fragile. If we can get rid of it, the procedure becomes more
> robust, I think.
Sure, you're right.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> Is there any way to tell from the WAL segments if they contain the full > page data? If so could you verify this on the second slave when it is > brought up? Or can you track this on the first slave and produce an > error in either pg_start_backup or pg_stop_backup() Sure. I will make a patch with the way to tell from the WAL segments if they contain the full page data. > I see in xlog.h XLR_BKP_REMOVABLE, the comment above it says that this > flag is used to indicate that the archiver can compress the full page > blocks to non-full page blocks. I am not familiar with where in the code > this actually happens but will this cause issues if the first standby is > processing WAL files from the archive? I confirmed the flag in xlog.c, so I seemed to only insert it in XLogInsert(). I consider whether it is available. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> >> > * Not correspond yet > >> > > >> > ?* full_page_write = off > >> > ? ?-> If the primary is "full_page_write = off", archive recovery may > >> > not act > >> > ? ? ? normally. Therefore the standby may need to check whether > >> > "full_page_write > >> > ? ? ? = off" to WAL. > >> > >> Isn't having a standby make the full_page_write = on in all case > >> (bypass configuration) ? > > > > what's the meaning? > > Yeah. full_page_writes is a WAL generation parameter. Standbys don't > generate WAL. I think you just have to insist that the master has it > on. Thanks. This has the following two problems. * pg_start_backup() must set 'on' to full_page_writes of the master that is actual writing of the WAL, but not the standby. * The standby doesn't need to connect to the master that's actual writing WAL. (Ex. Standby2 in Cascade Replication: Master - Standby1 - Standby2) I'm worried how I should clear these problems. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> >> > * Not correspond yet > >> > > >> > ?* full_page_write = off > >> > ? ?-> If the primary is "full_page_write = off", archive recovery may > >> > not act > >> > ? ? ? normally. Therefore the standby may need to check whether > >> > "full_page_write > >> > ? ? ? = off" to WAL. > >> > >> Isn't having a standby make the full_page_write = on in all case > >> (bypass configuration) ? > > > > what's the meaning? Thanks. This has the following two problems. * pg_start_backup() must set 'on' to full_page_writes of the master that is actual writing of the WAL, but not the standby. * The standby doesn't need to connect to the master that's actual writing WAL. (Ex. Standby2 in Cascade Replication: Master - Standby1 - Standby2) I'm worried how I should clear these problems. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > * Not correspond yet > > > > * full_page_write = off > >-> If the primary is "full_page_write = off", archive recovery may not > > act > > normally. Therefore the standby may need to check whether > > "full_page_write > > = off" to WAL. > > Isn't having a standby make the full_page_write = on in all case > (bypass configuration) ? what's the meaning? Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Online base backup from the hot-standby
> I will provide a patch which can exeute pg_start/stop_backup
> including to solve above comment and conditions in next stage.
> Then please review.
done.
* Procedure
1. Call pg_start_backup('x') on the standby.
2. Take a backup of the data dir.
3. Call pg_stop_backup() on the standby.
4. Copy the control file on the standby to the backup.
5. Check whether the control file is status during hot standby with
pg_controldata.
-> If the standby promote between 3. and 4., the backup can not recovery.
-> pg_control is that "Minimum recovery ending location" is equals 0/0.
-> backup-end record is not written.
* Not correspond yet
* full_page_write = off
-> If the primary is "full_page_write = off", archive recovery may not act
normally. Therefore the standby may need to check whether
"full_page_write
= off" to WAL.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_05.patch
Description: Binary data
standby_online_backup_doc.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> This version of the patch adds a field into pg_controldata that tries to
> store the source of the base backup while in recovery mode.
> I think your ultimate goal with this patch is to be able to take a
> backup of a running hot-standby slave and recover it as another
> instance. This patch seems to provide the ability to have the second
> slave stop recovery at minRecoveryPoint from the control file.
>
>
> My understanding of the procedure you want to get to to take base
> backups off a slave is
>
> 1. execute pg_start_backup('x') on the slave (*)
> 2. take a backup of the data dir
> 3. call pg_stop_backup() on the slave
> 4. Copy the control file on the slave
>
> This patch only addresses the recovery portions.
Yes.
> I don't think the above comment is very clear on what backupserver is.
> Perhaps
>
> /**
> * backupserver is used while postgresql is in recovery mode to
> * store the location of where the backup comes from.
> * When Postgres starts recovery operations
> * it is set to "none". During recovery it is updated to either "master",
> or "slave"
> * When recovery operations finish it is updated back to "none"
> **/
Done.
> Also shouldn't backupServer be the enum type of 'BackupServer' not int?
> Other enums in the structure such as DBState are defined this way.
Now, this is a same as wal_level, not DBState. No?
> Since I can't yet call pg_start_backup or pg_stop_backup() on the slave
> I am calling them on the master.
> (I also did some testing where I didn't put the system into backup
> mode). I admit that I am not sure what to look for as an indication that
> the system isn't recovering to the correct point. In much of my testing
> I was just verifying that the slave started and my data 'looked' okay.
Updated patch as can execute pg_start/stop_backup() on standby server.
One-pass of above steps(from 1. to 4.) is now done on this.
However, there are conditions.
* Master's full_page_write = on.
* On the slave, do not execute stop/promote operation before pg_stop_backup()
is executed.
* the result of pg_start_backup() may exceed the result of pg_stop_backup().
> I seem to get this warning in my logs when I start up the instance based
> on the slave backup.
> LOG: 0: database system was interrupted while in recovery at log
> time 2011-07-08 18:40:20 EDT
> HINT: If this has occurred more than once some data might be corrupted
> and you might need to choose an earlier recovery target
>
> I'm wondering if this warning is a bit misleading to users because it is
> an expected message when starting up an instance based on a slave backup
> (because the slave was already in recovery mode). If I shutdown this
> instance and start it up again I keep getting the warning. My
> understanding of your patch is that there shouldn't be any risk of
> corruption in that case (assuming your patch has no bugs). Can/should we
> be suppressing this message when we detect that we are recovering from a
> slave backup?
This has not been supported yet.
I do not see what state of this message.
Always happens when backup is taken from slave.
What do you think about an approach to add context, "unless take backup from
slave"?
> The direction of the patch has changed a bit during this commit fest. I
> think it would be good to provide an update on the rest of the changes
> you plan for this to be a complete useable feature. That would make it
> easier to comment on something you
> missed versus something your planning on dealing with in the next stage.
I see.
I will provide a patch which can exeute pg_start/stop_backup
including to solve above comment and conditions in next stage.
Then please review.
I change this patch status to "Returned with feedback".
Regards.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
standby_online_backup_04.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> As you proposed, adding new field which stores the backup end location > taken from minRecoveryPoint, into pg_control sounds good idea. Update patch. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_03.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> What about using backupStartPoint to check whether this recovery > started from the backup or not? No, postgres can check whether this recovery started from the backup or not, but can not check whether standby server or master (got backup from). Once recovery started, backupStartPoint is recorded to pg_control until recovery reaches backup end location, it is not related to any backup server. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> When the standby restarts after it crashes during recovery, it always > checks whether recovery has reached the backup end location by > using minRecoveryPoint even though the standby doesn't start from > the backup. This looks odd. Certainly. But, in this case, the state before recovery starts is lost. Therefore, postgres can not see that the backup got from whether standby server or master. What should? Should use pg_control? Ex. * Add 'Where to get backup' to pg_control. (default 'none') * When recovery starts, it checks it whether 'none'. * When minRecoveryPoint equals 0/0, change 'master'. * When minRecoveryPoint do not equals 0/0, change 'slave'. * When it reached the end of recovery, change 'none' . > - XLogRecPtrIsInvalid(ControlFile->backupStartPoint)) > + (XLogRecPtrIsInvalid(ControlFile->backupStartPoint) || > + reachedControlMinRecoveryPoint == true)) > The flag 'reachedControlMinRecoveryPoint' is really required? When recovery > reaches minRecoveryPoint, ControlFile->backupStartPoint is reset to zero. So > we can check whether recovery has reached minRecoveryPoint or not by only > doing XLogRecPtrIsInvalid(ControlFile->backupStartPoint). No? Yes. 'reachedControlMinRecoveryPoint' is unnecessary. > We should check if recovery has reached minRecoveryPoint before calling > CheckRecoveryConsistency() after reading new WAL record? Otherwise, > even if recovery has reached minRecoveryPoint, the standby cannot think > that it's in consistent state until it reads new WAL record. This is a same sequence with a case of backup end location. It should be no changed. > + if > (XLByteLT(ControlFile->minRecoveryPoint, EndRecPtr)) > + > ControlFile->minRecoveryPoint = EndRecPtr; > Why does ControlFile->minRecoveryPoint need to be set to EndRecPtr? Yes. I delete it. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> On this commitfest, the goal of the patch is to be able to be > recovered using "Minimum recovery ending location" in the control file. Done. Regards. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_02.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> > > Process of online base backup on standby server:
> > > 1. pg_start_backup('x');
> > > 2. copy the data directory
> > > 3. copy *pg_control*
> >
> > Who deletes the backup_label file created by pg_start_backup()?
> > Isn't pg_stop_backup() required to do that?
>
> You need it to take the system out of backup mode as well, don't you?
Certainly so.
Add to the above process:
4. pg_stop_backup();
But I do not consider a case such as to promote in backup mode yet.
I need to think a little, including it.
On this commitfest, the goal of the patch is to be able to be
recovered using "Minimum recovery ending location" in the control file.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> Considering everything that has been discussed on this thread so far.
>
> Do you still think your patch is the best way to accomplish base backups
> from standby servers?
> If not what changes do you think should be made?
I reconsider the way to not use pg_stop_backup().
Process of online base backup on standby server:
1. pg_start_backup('x');
2. copy the data directory
3. copy *pg_control*
Behavior while restore:
* read "Minimum recovery ending location" of the copied pg_control.
* use the value with the same purposes as the end-of-backup location.
-> When the value is equal to 0/0, this behavior do not do.
This situation is to acquire backup from master server.
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> What I think he is proposing would not require using pg_stop_backup() > but you could also modify pg_stop_back() to work as well. > > What do you think of this idea? > > Do you see how the patch can be reworked to accomplish this? The logic that not use pg_stop_backup() would be difficult, because pg_stop_backup() is used to identify minRecoveryPoint. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> 1) Today you can do a backup by just calling pg_start_backup('x'); copy
> the data directory and
> pg_stop_backup(); You do not need to use pg_basebackup to create a
> backup. The solution you are proposing would require pg_basebackup to be
> used to build backups from standby servers.
YES.
> 2) If I run pg_basebackup but do not specify '-x' then no pg_xlog
> segments are included in the output. The relevant pg_xlog segments are
> in the archive from the master. I can see situations where you are
> already copying the archive to the remote site that the new standby will
> be created in so you don't want to have to copy the pg_xlog segments
> twice over your network.
No, I don't matter because of the same behavior as 9.1.
Please see the part of "Before:" of the following answer.
> What Heikki is proposing will work both when you aren't using
> pg_basebackup (as long the output of pg_stop_backup() is somehow
> captured in a way that it can be read) and will also work with
> pg_basebackup when '-x' isn't specified.
I receive this mail, so I notice I do wrong recognition to what
Heikki is proposing.
my recognition:
Before:
* I thought Heikki proposes, "Execute SQL(pg_start_backup('x'); copy
the data directory and pg_stop_backup();) from the standby server
to the primary server".
-> I disliked this.
Now:
* Heikki is proposing both No using pg_basebackup and Not specify -x.
So,
* Use the output of pg_stop_backup().
* Don't use backup history file.
he thinks.
Right?
Jun Ishizuka
NTT Software Corporation
TEL：045-317-7018
E-Mail: ishizuka@po.ntts.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> I still think that's headed in the wrong direction. > (http://archives.postgresql.org/pgsql-hackers/2011-05/msg01405.php) Please check these mails, and teach the reason for content of the wrong direction. (http://archives.postgresql.org/pgsql-hackers/2011-06/msg00209.php) (http://archives.postgresql.org/pgsql-hackers/2011-05/msg01566.php) Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Online base backup from the hot-standby
( Quotation from http://archives.postgresql.org/pgsql-hackers/2011-05/msg01396.php ) > STEP1: Make startup process to acquire backup-end-position from > not only backup-end record but also backup-history-file . > * startup process allows to acquire backup-end-position > from backup-history-file . I have created a patch to the above-mentioned content. Please check it. Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp standby_online_backup_01.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
>>> I don't much like that approach. The standby would need to be able to >>> write the backup history file to the archive at the end of backup, and >>> we'd have to reintroduce the code to fetch it from archive and, when >>> streaming, from the master. At the moment, the archiver doesn't even run >>> in the standby. >> >> Please teach the reason for "The standby would need to be able to write >> the backup history file to the archive at the end of backup" . >> (I'd like to know why "to only pg_xlog" is wrong .) > > If the backup history file is not archived, the postgres process won't > find it when you try to restore from the base backup. The new server has > no access to the standby's pg_xlog directory. Thanks for the answer . But , it sends the backup history file to pg_xlog of new server (=backup server) when pg_basebackup is executed to the standby server , and so I was going to create the patch of such logic . I think it don't become the above-mentioned movement . >> Because there is the opinion of "Cascade replication" , I don't want to >> realize the function with the method which the standby requests to execute >> it on the primary server . >> >> (The opinion of "Cascade replication"： >> http://archives.postgresql.org/pgsql-hackers/2011-05/msg01150.php) > >I don't see how this helps. Hypothesis: * Online base backup was realized with the method which the standby requests to execute it on the "primary server" . * "Cascade replication" was developed , and user is using it . (Ex. Primary -- Standby1 -- Standby2) Situation: (1) Standby2 executes pg_basebackup . (2) Then, Standby2 accesses Standby1 . (3) But, it fails, because Standby2's primary is Standby1, not Primary . Result: * I don't want to realize the function with the method which the standby requests to execute it on the primary server . Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Online base backup from the hot-standby
> I don't much like that approach. The standby would need to be able to > write the backup history file to the archive at the end of backup, and > we'd have to reintroduce the code to fetch it from archive and, when > streaming, from the master. At the moment, the archiver doesn't even run > in the standby. Please teach the reason for "The standby would need to be able to write the backup history file to the archive at the end of backup" . (I'd like to know why "to only pg_xlog" is wrong .) Because there is the opinion of "Cascade replication" , I don't want to realize the function with the method which the standby requests to execute it on the primary server . (The opinion of "Cascade replication"： http://archives.postgresql.org/pgsql-hackers/2011-05/msg01150.php) Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Online base backup from the hot-standby
Hi I would like to develop function for 'Online base backup from the hot-standby' in PostgreSQL 9.2. Todo : Allow hot file system backups on standby servers (http://wiki.postgresql.org/wiki/Todo) [GOAL] * Make pg_basebackup to execute to the hot-standby server and acquire online-base-backup . - pg_basebackup can be executed to only primary server in PostgreSQL 9.1 . - But physical-copy(etc) under processing of pg_basebackup raises the load of primary server . - Therefore , this function is necessary . [Problem] (There is the following problems when hot-standby acquires online-base-backup like executing pg_basebackup to the primary server .) * pg_start_backup() and pg_stop_backup() can't be executed to the hot-standby server . - hot-standby can't insert backup-end record to WAL-files and can't operate CHECKPOINT . - Because hot-standby can't write anything in WAL-files . * hot-standby can't send WAL-files to archive server. - when pg_stop_backup() is executed to the primary server , it waits for completing sending wal to archive server , but hot-standby can't do it. [Policy] (I create with the following Policy .) * This function doesn't affect primary server . - I don't adopt the way which "hot-standby requests primary to execute pg_basebackup" , because I think about many standbys is connected with a primary . [Approach] * When pg_basebackup is executed to the hot-standby server , it executes RESTARTPOINT instead of CHECKPOINT . backup_label is made from the RESTARTPOINT's results , and is sent to the designated backup server using pg_basebackup connection . * Instead of inserting backup-end record , hot-standby writes backup-end-position in backup-history-file and sends to the designated backup server using pg_basebackup connection . - In 9.1 , startup process knows backup-end-position from only backup-end record . In addition to its logic, startup process can know backup-end-position from backup-history-file . As a result , startup process can recovery certainly without backup-end record . [Precondition] (As a result of the above-mentioned Policy and Approach , there is the following restrictions .) * Immediately after backup starting of WAL must contain full page writes . But the above-mentioned Approach can't satisfy the restriction according to circumstances . Because full_page_writes of primary might equal 'off' . When standby recovery WAL which is removed full page writes by pg_lesslog , it is the same . * Because recovery starts from last CHECKPOINT , it becomes long . * I has not thought new process that become taking the place of waiting for completing sending wal to archive server , yet. [Working Step] STEP1: Make startup process to acquire backup-end-position from not only backup-end record but also backup-history-file . * startup process allows to acquire backup-end-position from backup-history-file . * When pg_basebackup is executed , backup-history-file is sent to the designated backup server . STEP2: Make pg_start_backup() and pg_stop_backup() to be executed by the hot-standby server. [Action until The first CommitFest (on June 15)] I will create a patch to STEP1 . (The patch will be able to settle a problem of Omnipitr-backup-slave.) (a problem of Omnipitr-backup-slave : http://archives.postgresql.org/pgsql-hackers/2011-03/msg01490.php) * Shedule of creating STEP2 is the next CommitFest (in September 15) Jun Ishizuka NTT Software Corporation TEL：045-317-7018 E-Mail: ishizuka@po.ntts.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
