Re: replay pause vs. standby promotion
Hello > If we don't ignore walreceiver and does try to connect to the master, > a promotion and recovery cannot end forever since new WAL data can > be streamed. You think this behavior is more consistent? We have no simple point to stop replay. Well, except for "immediately" - just one easy stop. But I agree that this is not the best option. Simple and clear, but not best one for data when we want to replay as much as possible from archive. > IMO it's valid to replay all the WAL data available to avoid data loss > before a promotion completes. But in case of still working primary (with archive_command) we choose quite random time to promote. A random time when the primary did not save the new wal segment. or even when a temporary error of restore_command occurs? We mention just cp command in docs. I know users uses cp (e.g. from NFS) without further error handling. regards, Sergei
Re: replay pause vs. standby promotion
On 2020/03/25 19:42, Sergei Kornilov wrote: Hi Could we add a few words in func.sgml to clarify the behavior? Especially for users from my example above. Something like: If a promotion is triggered while recovery is paused, the paused state ends, replay of any WAL immediately available in the archive or in pg_wal will be continued and then a promotion will be completed. This description is true if pause is requested by pg_wal_replay_pause(), but not if recovery target is reached and pause is requested by recovery_target_action=pause. In the latter case, even if there are WAL data avaiable in pg_wal or archive, they are not replayed, i.e., the promotion completes immediately. Probably we should document those two cases explicitly to avoid the confusion about a promotion and recovery pause? This is description for pg_wal_replay_pause, but actually we suggest to call pg_wal_replay_resume in recovery_target_action=pause... So, I agree, we need to document both cases. PS: I think we have inconsistent behavior here... Read wal during promotion from local pg_wal AND call restore_command, but ignore walreceiver also seems strange for my DBA hat... If we don't ignore walreceiver and does try to connect to the master, a promotion and recovery cannot end forever since new WAL data can be streamed. You think this behavior is more consistent? IMO it's valid to replay all the WAL data available to avoid data loss before a promotion completes. Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
Hi >> Could we add a few words in func.sgml to clarify the behavior? Especially >> for users from my example above. Something like: >> >>> If a promotion is triggered while recovery is paused, the paused state >>> ends, replay of any WAL immediately available in the archive or in pg_wal >>> will be continued and then a promotion will be completed. > > This description is true if pause is requested by pg_wal_replay_pause(), > but not if recovery target is reached and pause is requested by > recovery_target_action=pause. In the latter case, even if there are WAL data > avaiable in pg_wal or archive, they are not replayed, i.e., the promotion > completes immediately. Probably we should document those two cases > explicitly to avoid the confusion about a promotion and recovery pause? This is description for pg_wal_replay_pause, but actually we suggest to call pg_wal_replay_resume in recovery_target_action=pause... So, I agree, we need to document both cases. PS: I think we have inconsistent behavior here... Read wal during promotion from local pg_wal AND call restore_command, but ignore walreceiver also seems strange for my DBA hat... regards, Sergei
Re: replay pause vs. standby promotion
On 2020/03/25 0:17, Sergei Kornilov wrote: Hello I pushed the latest version of the patch. If you have further opinion about immediate promotion, let's keep discussing that! Thank you! Honestly, I forgot that the promotion is documented in high-availability.sgml as: Before failover, any WAL immediately available in the archive or in pg_wal will be restored, but no attempt is made to connect to the master. I mistakenly thought that promote should be "immediately"... If a promotion is triggered while recovery is paused, the paused state ends and a promotion continues. Could we add a few words in func.sgml to clarify the behavior? Especially for users from my example above. Something like: If a promotion is triggered while recovery is paused, the paused state ends, replay of any WAL immediately available in the archive or in pg_wal will be continued and then a promotion will be completed. This description is true if pause is requested by pg_wal_replay_pause(), but not if recovery target is reached and pause is requested by recovery_target_action=pause. In the latter case, even if there are WAL data avaiable in pg_wal or archive, they are not replayed, i.e., the promotion completes immediately. Probably we should document those two cases explicitly to avoid the confusion about a promotion and recovery pause? Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
Hello > I pushed the latest version of the patch. If you have further opinion > about immediate promotion, let's keep discussing that! Thank you! Honestly, I forgot that the promotion is documented in high-availability.sgml as: > Before failover, any WAL immediately available in the archive or in pg_wal > will be > restored, but no attempt is made to connect to the master. I mistakenly thought that promote should be "immediately"... > If a promotion is triggered while recovery is paused, the paused state ends > and a promotion continues. Could we add a few words in func.sgml to clarify the behavior? Especially for users from my example above. Something like: > If a promotion is triggered while recovery is paused, the paused state ends, > replay of any WAL immediately available in the archive or in pg_wal will be > continued and then a promotion will be completed. regards, Sergei
Re: replay pause vs. standby promotion
On 2020/03/24 0:57, Fujii Masao wrote: On 2020/03/24 0:17, Sergei Kornilov wrote: Hello You meant that the promotion request should cause the recovery to finish immediately even if there are still outstanding WAL records, and cause the standby to become the master? Oh, I get your point. But yes, I expect that in case of promotion request during a pause, the user (me too) will want to have exactly the current state, not latest available in WALs. Basically I'd like the promotion to make the standby replay all the WAL even if it's requested during pause state. OTOH I understand there are use cases where immediate promotion is useful, as you explained. So, +1 to add something like "pg_ctl promote -m immediate". But I'm afraid that now it's too late to add such feature into v13. Probably it's an item for v14 I pushed the latest version of the patch. If you have further opinion about immediate promotion, let's keep discussing that! Also we need to go back to the original patch posted at [1]. [1] https://www.postgresql.org/message-id/flat/19168211580382...@myt5-b646bde4b8f3.qloud-c.yandex.net Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
On 2020/03/24 0:17, Sergei Kornilov wrote: Hello You meant that the promotion request should cause the recovery to finish immediately even if there are still outstanding WAL records, and cause the standby to become the master? Oh, I get your point. But yes, I expect that in case of promotion request during a pause, the user (me too) will want to have exactly the current state, not latest available in WALs. Basically I'd like the promotion to make the standby replay all the WAL even if it's requested during pause state. OTOH I understand there are use cases where immediate promotion is useful, as you explained. So, +1 to add something like "pg_ctl promote -m immediate". But I'm afraid that now it's too late to add such feature into v13. Probably it's an item for v14 Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
On 2020/03/23 23:55, Robert Haas wrote: On Mon, Mar 23, 2020 at 10:36 AM Fujii Masao wrote: If we would like to have the promotion method to finish recovery immediately, IMO we should implement something like "pg_ctl promote -m fast". That is, we need to add new method into the promotion. I think 'immediate' would be a better choice. One reason is that we've used the term 'fast promotion' in the past for a different feature. Another is that 'immediate' might sound slightly scary to people who are familiar with what 'pg_ctl stop -mimmediate' does. And you want people doing this to be just a little bit scared: not too scared, but a little scared. +1 When I proposed the feature five years before, I used "immediate" as the option value. https://postgr.es/m/cahgqgwhtvydqkzaywya9zyylecakif5p0kpcpune_tsrgtf...@mail.gmail.com Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
Hello > You meant that the promotion request should cause the recovery > to finish immediately even if there are still outstanding WAL records, > and cause the standby to become the master? Oh, I get your point. But yes, I expect that in case of promotion request during a pause, the user (me too) will want to have exactly the current state, not latest available in WALs. Real usercase from my experience: The user wants to update a third-party application. In case of problems, he wants to return to the old version of the application and the unchanged replica. Thus, it sets a pause on standby and performs an update. If all is ok - he will resume replay. In case of some problems he plans to promote standby. But oops, standby will ignore promote signals during pause and we need get currect LSN from standby and restart it with recovery_target_lsn = ? and recovery_target_action = promote to achieve this state. regards, Sergei
Re: replay pause vs. standby promotion
On Mon, Mar 23, 2020 at 10:36 AM Fujii Masao wrote: > If we would like to have the promotion method to finish recovery > immediately, IMO we should implement something like > "pg_ctl promote -m fast". That is, we need to add new method into > the promotion. I think 'immediate' would be a better choice. One reason is that we've used the term 'fast promotion' in the past for a different feature. Another is that 'immediate' might sound slightly scary to people who are familiar with what 'pg_ctl stop -mimmediate' does. And you want people doing this to be just a little bit scared: not too scared, but a little scared. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: replay pause vs. standby promotion
On 2020/03/23 22:46, Sergei Kornilov wrote: Hello (I am trying to find an opportunity to review this patch...) Thanks for the review! It's really helpful! Consider test case with streaming replication: on primary: create table foo (i int); on standby: postgres=# select pg_wal_replay_pause(); pg_wal_replay_pause - (1 row) postgres=# select pg_is_wal_replay_paused(); pg_is_wal_replay_paused - t (1 row) postgres=# table foo; i --- (0 rows) Execute "insert into foo values (1);" on primary postgres=# select pg_promote (); pg_promote t (1 row) postgres=# table foo; i --- 1 And we did replay one additional change during promote. I think this is wrong behavior. Possible can be fixed by +if (PromoteIsTriggered()) break; /* Setup error traceback support for ereport() */ errcallback.callback = rm_redo_error_callback; You meant that the promotion request should cause the recovery to finish immediately even if there are still outstanding WAL records, and cause the standby to become the master? I don't think that it's the expected (also existing) behavior of the promotion. That is, the promotion request should cause the recovery to replay as much WAL records as possible, to the end, in order to avoid data loss. No? If we would like to have the promotion method to finish recovery immediately, IMO we should implement something like "pg_ctl promote -m fast". That is, we need to add new method into the promotion. Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
Hello (I am trying to find an opportunity to review this patch...) Consider test case with streaming replication: on primary: create table foo (i int); on standby: postgres=# select pg_wal_replay_pause(); pg_wal_replay_pause - (1 row) postgres=# select pg_is_wal_replay_paused(); pg_is_wal_replay_paused - t (1 row) postgres=# table foo; i --- (0 rows) Execute "insert into foo values (1);" on primary postgres=# select pg_promote (); pg_promote t (1 row) postgres=# table foo; i --- 1 And we did replay one additional change during promote. I think this is wrong behavior. Possible can be fixed by +if (PromoteIsTriggered()) break; /* Setup error traceback support for ereport() */ errcallback.callback = rm_redo_error_callback; regards, Sergei
Re: replay pause vs. standby promotion
On 2020/03/20 15:22, Atsushi Torikoshi wrote: On Fri, Mar 6, 2020 at 10:18 PM Fujii Masao mailto:masao.fu...@oss.nttdata.com>> wrote: OK, so patch attached. This patch causes, if a promotion is triggered while recovery is paused, the paused state to end and a promotion to continue. OTOH, this patch makes pg_wal_replay_pause() and _resume() throw an error if it's executed while a promotion is ongoing. Regarding recovery_target_action, if the recovery target is reached while a promotion is ongoing, "pause" setting will act the same as "promote", i.e., recovery will finish and the server will start to accept connections. To implement the above, I added new shared varible indicating whether a promotion is triggered or not. Only startup process can update this shared variable. Other processes like read-only backends can check whether promotion is ongoing, via this variable. I added new function PromoteIsTriggered() that returns true if a promotion is triggered. Since the name of this function and the existing function IsPromoteTriggered() are confusingly similar, I changed the name of IsPromoteTriggered() to IsPromoteSignaled, as more appropriate name. I've confirmed the patch works as you described above. And I also poked around it a little bit but found no problems. Thanks for the review! Barrying any objection, I will commit the patch. Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters
Re: replay pause vs. standby promotion
On Fri, Mar 6, 2020 at 10:18 PM Fujii Masao wrote: > > OK, so patch attached. > > This patch causes, if a promotion is triggered while recovery is paused, > the paused state to end and a promotion to continue. OTOH, this patch > makes pg_wal_replay_pause() and _resume() throw an error if it's executed > while a promotion is ongoing. Regarding recovery_target_action, if the recovery target is reached > while a promotion is ongoing, "pause" setting will act the same as > "promote", > i.e., recovery will finish and the server will start to accept connections. > > To implement the above, I added new shared varible indicating whether > a promotion is triggered or not. Only startup process can update this > shared > variable. Other processes like read-only backends can check whether > promotion is ongoing, via this variable. > > I added new function PromoteIsTriggered() that returns true if a promotion > is triggered. Since the name of this function and the existing function > IsPromoteTriggered() are confusingly similar, I changed the name of > IsPromoteTriggered() to IsPromoteSignaled, as more appropriate name. > I've confirmed the patch works as you described above. And I also poked around it a little bit but found no problems. Regards, -- Atsushi Torikoshi
Re: replay pause vs. standby promotion
On 2020/03/04 23:40, Jehan-Guillaume de Rorthais wrote: On Wed, 04 Mar 2020 15:00:54 +0300 Sergei Kornilov wrote: Hello I want to start this discussion because this is related to the patch (propoesd at the thread [1]) that I'm reviewing. It does that partially, i.e., prefers the promotion only when the pause is requested by recovery_target_action=pause. But I think that it's reasonable and more consistent to do that whether whichever the pause is requested by pg_wal_replay_pause() or recovery_target_action. +1. +1 And pg_wal_replay_pause () should probably raise an error explaining the standby ignores the pause because of ongoing promotion. OK, so patch attached. This patch causes, if a promotion is triggered while recovery is paused, the paused state to end and a promotion to continue. OTOH, this patch makes pg_wal_replay_pause() and _resume() throw an error if it's executed while a promotion is ongoing. Regarding recovery_target_action, if the recovery target is reached while a promotion is ongoing, "pause" setting will act the same as "promote", i.e., recovery will finish and the server will start to accept connections. To implement the above, I added new shared varible indicating whether a promotion is triggered or not. Only startup process can update this shared variable. Other processes like read-only backends can check whether promotion is ongoing, via this variable. I added new function PromoteIsTriggered() that returns true if a promotion is triggered. Since the name of this function and the existing function IsPromoteTriggered() are confusingly similar, I changed the name of IsPromoteTriggered() to IsPromoteSignaled, as more appropriate name. I'd like to apply the change of log message that Sergei proposed at [1] after commiting this patch if it's ok. [1] https://www.postgresql.org/message-id/flat/19168211580382...@myt5-b646bde4b8f3.qloud-c.yandex.net Regards, -- Fujii Masao NTT DATA CORPORATION Advanced Platform Technology Group Research and Development Headquarters diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index c1128f89ec..86726376f1 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -3570,6 +3570,9 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows This setting has no effect if no recovery target is set. If is not enabled, a setting of pause will act the same as shutdown. +If the recovery target is reached while a promotion is ongoing, +a setting of pause will act the same as +promote. In any case, if a recovery target is configured but the archive diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 323366feb6..bcd456f52d 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -20154,6 +20154,13 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup()); recovery is resumed. + +pg_wal_replay_pause and +pg_wal_replay_resume cannot be executed while +a promotion is ongoing. If a promotion is triggered while recovery +is paused, the paused state ends and a promotion continues. + + If streaming replication is disabled, the paused state may continue indefinitely without problem. While streaming replication is in diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 4361568882..c545aeffa3 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -229,6 +229,12 @@ static bool LocalRecoveryInProgress = true; */ static bool LocalHotStandbyActive = false; +/* + * Local copy of SharedPromoteIsTriggered variable. False actually means "not + * known, need to check the shared state". + */ +static bool LocalPromoteIsTriggered = false; + /* * Local state for XLogInsertAllowed(): * 1: unconditionally allowed to insert XLOG @@ -654,6 +660,12 @@ typedef struct XLogCtlData */ boolSharedHotStandbyActive; + /* +* SharedPromoteIsTriggered indicates if a standby promotion has been +* triggered. Protected by info_lck. +*/ + boolSharedPromoteIsTriggered; + /* * WalWriterSleeping indicates whether the WAL writer is currently in * low-power mode (and hence should be nudged if an async commit occurs). @@ -912,6 +924,7 @@ static void InitControlFile(uint64 sysidentifier); static void WriteControlFile(void); static void ReadControlFile(void); static char *str_time(pg_time_t tnow); +static void SetPromoteIsTriggered(void); static bool CheckForStandbyTrigger(void); #ifdef WAL_DEBUG @@ -5112,6 +5125,7 @@ XLOGShmemInit(void) XLogCtl->XLogCacheBlck = XLOGbuffers - 1; XLogCtl->SharedRecoveryInProgress = true; XLogCtl->SharedHotStandbyActive = false; + XLogCtl->SharedPromoteIsTriggered = false; XLogCtl->WalWriterSleeping = false;
Re: replay pause vs. standby promotion
On Wed, 04 Mar 2020 15:00:54 +0300 Sergei Kornilov wrote: > Hello > > > I want to start this discussion because this is related to the patch > > (propoesd at the thread [1]) that I'm reviewing. It does that partially, > > i.e., prefers the promotion only when the pause is requested by > > recovery_target_action=pause. But I think that it's reasonable and > > more consistent to do that whether whichever the pause is requested > > by pg_wal_replay_pause() or recovery_target_action. > > +1. +1 And pg_wal_replay_pause () should probably raise an error explaining the standby ignores the pause because of ongoing promotion.
Re: replay pause vs. standby promotion
Hello > I want to start this discussion because this is related to the patch > (propoesd at the thread [1]) that I'm reviewing. It does that partially, > i.e., prefers the promotion only when the pause is requested by > recovery_target_action=pause. But I think that it's reasonable and > more consistent to do that whether whichever the pause is requested > by pg_wal_replay_pause() or recovery_target_action. +1. I'm just not sure if this is safe for replay logic, so I did not touch this behavior in my proposal. (hmm, I wanted to mention this, but apparently forgot) regards, Sergei
