On Fri, Oct 2, 2026 at 12:27 AM Bohyun Lee <[email protected]> wrote:
> 2. Handoff atomicity and shutdown ordering > > Feedback: the old primary might continue processing transactions after the > handoff. The handoff command should shut down the primary at the handoff > record. > The handoff could also be represented by the shutdown checkpoint itself. Marking the shutdown checkpoint is precisely the approach I've taken in my own work, so I've attached two small patches to show this. They were extracted from a larger series that's not quite ready to share, so some details may need to change. There are a couple forward-looking bits that are out of context, but it's mostly independent of the rest. I didn't share earlier because I didn't think it was quite compatible with your proposal, but since you mentioned it, I'm now curious how close it is to being compatible. It's worth calling out the fact that the old binary must be aware of the "handoff" (my series calls it a "version boundary" or "upgrade boundary", but it seems to be the same concept). In my view, whatever the old binary needs is as much as we can commit for PG20, and anything in the target version should probably not be committed until PG21 at the earliest. They have to be developed together, so timing is a bit tricky, but using the shutdown checkpoint seems to me the least invasive way to be forward-looking. Your first patch (I haven't looked at the second) didn't distinguish the needs of the old and new binaries cleanly since it was a single patch. (BTW, 0002 conventionally means the second patch in a series, applied on top of 0001 in order. You can make git add patch numbers and version numbers with `git format-patch master -v 2`) -- John Naylor Amazon Web Services
From 9074b812f4060e22683d18c38bc34fc234050cad Mon Sep 17 00:00:00 2001 From: John Naylor <[email protected]> Date: Mon, 20 Jul 2026 17:15:11 -0400 Subject: [PATCH v20261003 1/2] Allow marking a shutdown checkpoint as an upgrade boundary A major version upgrade that keeps standbys current by replaying WAL, instead of rebuilding them, needs a well-defined end to the old server's WAL: the final shutdown checkpoint written by the old binary. Standbys must replay up to that point and go no further, because whatever follows is written by the new version. pg_request_upgrade_boundary() records an upgrade request in shared memory; the next shutdown checkpoint consumes it and records itself as an upgrade boundary in a new CheckPoint field. pg_cancel_upgrade_boundary() withdraws a pending request, and pg_upgrade_boundary_requested() reports the current state. The request intentionally lives only in shared memory: a crash writes no shutdown checkpoint, so it forgets the request. Online checkpoints and end-of-recovery checkpoints are never marked. The flag is part of the checkpoint copy in pg_control, so pg_controldata reports it after shutdown. pg_waldump shows it in the checkpoint record's description. pg_resetwal clears the flag, since it discards the WAL that contains the marked checkpoint. TODO: pg_control_checkpoint() Nothing acts on the marking yet; a follow-up commit will teach standbys to pause when encountering it. XXX: bump PG_CONTROL_VERSION --- doc/src/sgml/func/func-admin.sgml | 92 +++++++++++++++++ src/backend/access/rmgrdesc/xlogdesc.c | 26 +++++ src/backend/access/transam/xlog.c | 67 ++++++++++++- src/backend/access/transam/xlogfuncs.c | 58 +++++++++++ src/bin/pg_controldata/pg_controldata.c | 19 ++++ src/bin/pg_resetwal/pg_resetwal.c | 6 ++ src/include/access/xlog.h | 4 + src/include/catalog/pg_control.h | 18 ++++ src/include/catalog/pg_proc.dat | 13 +++ src/test/recovery/meson.build | 1 + src/test/recovery/t/057_upgrade_boundary.pl | 105 ++++++++++++++++++++ src/tools/pgindent/typedefs.list | 1 + 12 files changed, 407 insertions(+), 3 deletions(-) create mode 100644 src/test/recovery/t/057_upgrade_boundary.pl diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 64b0e7bb972..5fb55b1e14f 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -888,6 +888,98 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset </sect2> + <sect2 id="functions-upgrade-boundary"> + <title>Upgrade Boundary Functions</title> + + <para> + The functions shown in <xref + linkend="functions-upgrade-boundary-table"/> control the marking of an + <firstterm>upgrade boundary</firstterm>: a shutdown checkpoint that is + the deliberate end of the current major version's write-ahead log, + written in preparation for an in-place major version upgrade. + </para> + + <table id="functions-upgrade-boundary-table"> + <title>Upgrade Boundary Functions</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="func_table_entry"><para role="func_signature"> + Function + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_request_upgrade_boundary</primary> + </indexterm> + <function>pg_request_upgrade_boundary</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Requests that the next shutdown checkpoint be marked as an upgrade + boundary. The request only takes effect at a smart or fast + shutdown; it does not survive a crash, and it does not affect + checkpoints of a running server. This function cannot be executed + during recovery. + </para> + <para> + This function is restricted to superusers by default, but other users + can be granted EXECUTE to run the function. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_cancel_upgrade_boundary</primary> + </indexterm> + <function>pg_cancel_upgrade_boundary</function> () + <returnvalue>void</returnvalue> + </para> + <para> + Cancels a requested upgrade boundary. It is not an error to call + this when no boundary has been requested. This function cannot be + executed during recovery. + </para> + <para> + This function is restricted to superusers by default, but other users + can be granted EXECUTE to run the function. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_upgrade_boundary_requested</primary> + </indexterm> + <function>pg_upgrade_boundary_requested</function> () + <returnvalue>boolean</returnvalue> + </para> + <para> + Returns true if an upgrade boundary has been requested for the next + shutdown checkpoint. During recovery this always returns + <literal>false</literal>. + </para></entry> + </row> + </tbody> + </tgroup> + </table> + + <para> + If a server that was shut down at an upgrade boundary is simply started + again with the same major version, the boundary is cancelled: writing + new write-ahead log on the same timeline supersedes it. + </para> + + </sect2> + <sect2 id="functions-snapshot-synchronization"> <title>Snapshot Synchronization Functions</title> diff --git a/src/backend/access/rmgrdesc/xlogdesc.c b/src/backend/access/rmgrdesc/xlogdesc.c index 64e749c1e6b..b227da79e63 100644 --- a/src/backend/access/rmgrdesc/xlogdesc.c +++ b/src/backend/access/rmgrdesc/xlogdesc.c @@ -55,6 +55,27 @@ get_wal_level_string(int wal_level) return wal_level_str; } +/* + * Find a string representation for a checkpoint's upgrade flag + */ +static const char * +get_upgrade_flag_string(uint8 flag) +{ + switch (flag) + { + case UPGRADE_FLAG_NONE: + return "none"; + case UPGRADE_FLAG_BOUNDARY: + return "boundary"; + case UPGRADE_FLAG_START: + return "start"; + case UPGRADE_FLAG_GOLIVE: + return "go-live"; + } + + return "?"; +} + const char * get_checksum_state_string(uint32 state) { @@ -126,6 +147,11 @@ xlog_desc(StringInfo buf, XLogReaderState *record) checkpoint->oldestActiveXid, get_checksum_state_string(checkpoint->dataChecksumState), (info == XLOG_CHECKPOINT_SHUTDOWN) ? "shutdown" : "online"); + + /* Only upgrade machinery writes a flag, so keep quiet otherwise */ + if (checkpoint->upgradeFlag != UPGRADE_FLAG_NONE) + appendStringInfo(buf, "; upgrade flag %s", + get_upgrade_flag_string(checkpoint->upgradeFlag)); } else if (info == XLOG_NEXTOID) { diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 9ec0be77ca0..83bba8220c5 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -538,6 +538,13 @@ typedef struct XLogCtlData */ bool WalWriterSleeping; + /* + * upgradeBoundaryRequested indicates that the next shutdown checkpoint is + * to be marked as a major version upgrade boundary. Protected by + * info_lck. + */ + bool upgradeBoundaryRequested; + /* * During recovery, we keep a copy of the latest checkpoint record here. * lastCheckPointRecPtr points to start of checkpoint record and @@ -6161,6 +6168,17 @@ StartupXLOG(void) errmsg("control file contains invalid database cluster state"))); } + /* + * If the last shutdown checkpoint was marked as an upgrade boundary but + * we are starting up as usual, no upgrade happened. The boundary is + * cancelled as a side effect of writing new WAL on this timeline. + */ + if (ControlFile->state == DB_SHUTDOWNED && + ControlFile->checkPointCopy.upgradeFlag == UPGRADE_FLAG_BOUNDARY) + ereport(LOG, + (errmsg("database system was shut down at an upgrade boundary"), + errdetail("Starting up normally cancels the upgrade boundary."))); + /* This is just to allow attaching to startup process with a debugger */ #ifdef XLOG_REPLAY_DELAY if (ControlFile->state != DB_SHUTDOWNED) @@ -7374,6 +7392,36 @@ GetLastSegSwitchData(XLogRecPtr *lastSwitchLSN) return result; } +/* + * Request or cancel an upgrade boundary: when requested, the next shutdown + * checkpoint is marked as an upgrade boundary. + * + * The request lives only in shared memory: a crash, which writes no + * shutdown checkpoint, forgets it. + */ +void +SetUpgradeBoundaryRequest(bool request) +{ + SpinLockAcquire(&XLogCtl->info_lck); + XLogCtl->upgradeBoundaryRequested = request; + SpinLockRelease(&XLogCtl->info_lck); +} + +/* + * Report whether an upgrade boundary is currently requested. + */ +bool +UpgradeBoundaryActive(void) +{ + bool active; + + SpinLockAcquire(&XLogCtl->info_lck); + active = XLogCtl->upgradeBoundaryRequested; + SpinLockRelease(&XLogCtl->info_lck); + + return active; +} + /* * This must be called ONCE during postmaster or standalone-backend shutdown */ @@ -7408,6 +7456,8 @@ ShutdownXLOG(int code, Datum arg) CreateRestartPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_FAST); else { + int flags = CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_FAST; + /* * If archiving is enabled, rotate the last XLOG file so that all the * remaining records are archived (postmaster wakes up the archiver @@ -7417,7 +7467,11 @@ ShutdownXLOG(int code, Datum arg) if (XLogArchivingActive()) RequestXLogSwitch(false); - CreateCheckPoint(CHECKPOINT_IS_SHUTDOWN | CHECKPOINT_FAST); + /* Mark the shutdown checkpoint as an upgrade boundary if requested. */ + if (UpgradeBoundaryActive()) + flags |= CHECKPOINT_UPGRADE_BOUNDARY; + + CreateCheckPoint(flags); } } @@ -7430,7 +7484,7 @@ CheckpointFlagsString(int flags) { static char buf[128]; - snprintf(buf, sizeof(buf), "%s%s%s%s%s%s%s%s", + snprintf(buf, sizeof(buf), "%s%s%s%s%s%s%s%s%s", (flags & CHECKPOINT_IS_SHUTDOWN) ? " shutdown" : "", (flags & CHECKPOINT_END_OF_RECOVERY) ? " end-of-recovery" : "", (flags & CHECKPOINT_FAST) ? " fast" : "", @@ -7438,7 +7492,8 @@ CheckpointFlagsString(int flags) (flags & CHECKPOINT_WAIT) ? " wait" : "", (flags & CHECKPOINT_CAUSE_XLOG) ? " wal" : "", (flags & CHECKPOINT_CAUSE_TIME) ? " time" : "", - (flags & CHECKPOINT_FLUSH_UNLOGGED) ? " flush-unlogged" : ""); + (flags & CHECKPOINT_FLUSH_UNLOGGED) ? " flush-unlogged" : "", + (flags & CHECKPOINT_UPGRADE_BOUNDARY) ? " upgrade-boundary" : ""); return buf; } @@ -7948,6 +8003,12 @@ CreateCheckPoint(int flags) checkPoint.logicalDecodingEnabled = IsLogicalDecodingEnabled(); + if ((flags & CHECKPOINT_IS_SHUTDOWN) != 0 && + (flags & CHECKPOINT_UPGRADE_BOUNDARY) != 0) + checkPoint.upgradeFlag = UPGRADE_FLAG_BOUNDARY; + else + checkPoint.upgradeFlag = UPGRADE_FLAG_NONE; + MultiXactGetCheckptMulti(shutdown, &checkPoint.nextMulti, &checkPoint.nextMultiOffset, diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 1f52bf7b420..bffd724b473 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -222,6 +222,64 @@ pg_switch_wal(PG_FUNCTION_ARGS) PG_RETURN_LSN(switchpoint); } +/* + * pg_request_upgrade_boundary: request an upgrade boundary + * + * When requested, the next shutdown checkpoint is marked as an upgrade + * boundary. + * + * Permission checking for this function is managed through the normal + * GRANT system. + */ +Datum +pg_request_upgrade_boundary(PG_FUNCTION_ARGS) +{ + if (RecoveryInProgress()) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("recovery is in progress"), + errhint("WAL control functions cannot be executed during recovery."))); + + SetUpgradeBoundaryRequest(true); + + PG_RETURN_VOID(); +} + +/* + * pg_cancel_upgrade_boundary: cancel a requested upgrade boundary + * + * Permission checking for this function is managed through the normal + * GRANT system. + */ +Datum +pg_cancel_upgrade_boundary(PG_FUNCTION_ARGS) +{ + if (RecoveryInProgress()) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("recovery is in progress"), + errhint("WAL control functions cannot be executed during recovery."))); + + SetUpgradeBoundaryRequest(false); + + PG_RETURN_VOID(); +} + +/* + * pg_upgrade_boundary_requested: report whether an upgrade boundary is active + * + * During recovery there is no requesting a boundary, so simply return + * false rather than raising an error. + */ +Datum +pg_upgrade_boundary_requested(PG_FUNCTION_ARGS) +{ + if (RecoveryInProgress()) + PG_RETURN_BOOL(false); + + PG_RETURN_BOOL(UpgradeBoundaryActive()); +} + /* * pg_log_standby_snapshot: call LogStandbySnapshot() * diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c index 6a0f848d8d0..cfea8d7325c 100644 --- a/src/bin/pg_controldata/pg_controldata.c +++ b/src/bin/pg_controldata/pg_controldata.c @@ -84,6 +84,23 @@ wal_level_str(WalLevel wal_level) return _("unrecognized \"wal_level\""); } +static const char * +upgrade_flag_str(uint8 flag) +{ + switch (flag) + { + case UPGRADE_FLAG_NONE: + return "none"; + case UPGRADE_FLAG_BOUNDARY: + return "boundary"; + case UPGRADE_FLAG_START: + return "start"; + case UPGRADE_FLAG_GOLIVE: + return "go-live"; + } + return _("unrecognized upgrade flag"); +} + int main(int argc, char *argv[]) @@ -291,6 +308,8 @@ main(int argc, char *argv[]) ControlFile->checkPointCopy.newestCommitTsXid); printf(_("Latest checkpoint's data_checksum_version:%u\n"), ControlFile->checkPointCopy.dataChecksumState); + printf(_("Latest checkpoint's upgrade flag: %s\n"), + upgrade_flag_str(ControlFile->checkPointCopy.upgradeFlag)); printf(_("Time of latest checkpoint: %s\n"), ckpttime_str); printf(_("Fake LSN counter for unlogged rels: %X/%08X\n"), diff --git a/src/bin/pg_resetwal/pg_resetwal.c b/src/bin/pg_resetwal/pg_resetwal.c index 41afc4c1316..b3df37830ce 100644 --- a/src/bin/pg_resetwal/pg_resetwal.c +++ b/src/bin/pg_resetwal/pg_resetwal.c @@ -930,6 +930,12 @@ RewriteControlFile(void) */ ControlFile.data_checksum_lsn = InvalidXLogRecPtr; + /* + * The checkpoint being rewritten is no longer the one that was marked as + * an upgrade boundary, if any; the new empty WAL starts fresh. + */ + ControlFile.checkPointCopy.upgradeFlag = UPGRADE_FLAG_NONE; + /* * Force the defaults for max_* settings. The values don't really matter * as long as wal_level='minimal'; the postmaster will reset these fields diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index 7a590b7e1ea..23d788b427b 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -167,6 +167,8 @@ extern PGDLLIMPORT bool XLOG_DEBUG; #define CHECKPOINT_FAST 0x0004 /* Do it without delays */ #define CHECKPOINT_FORCE 0x0008 /* Force even if no activity */ #define CHECKPOINT_FLUSH_UNLOGGED 0x0010 /* Flush unlogged tables */ +#define CHECKPOINT_UPGRADE_BOUNDARY 0x0200 /* Mark shutdown checkpoint as an + * upgrade boundary */ /* These are important to RequestCheckpoint */ #define CHECKPOINT_WAIT 0x0020 /* Wait for completion */ #define CHECKPOINT_REQUESTED 0x0040 /* Checkpoint request has been made */ @@ -295,6 +297,8 @@ extern XLogRecPtr GetFlushRecPtr(TimeLineID *insertTLI); extern TimeLineID GetWALInsertionTimeLine(void); extern TimeLineID GetWALInsertionTimeLineIfSet(void); extern XLogRecPtr GetLastImportantRecPtr(void); +extern void SetUpgradeBoundaryRequest(bool request); +extern bool UpgradeBoundaryActive(void); extern void SetWalWriterSleeping(bool sleeping); diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h index c3c934d0012..2a5e37d47e6 100644 --- a/src/include/catalog/pg_control.h +++ b/src/include/catalog/pg_control.h @@ -66,8 +66,26 @@ typedef struct CheckPoint /* data checksums state at the time of the checkpoint */ uint32 dataChecksumState; + + + /* + * Role of this checkpoint during an in-place major version upgrade, or + * UPGRADE_FLAG_NONE otherwise. + */ + uint8 upgradeFlag; /* one of the UPGRADE_FLAG_* values below */ } CheckPoint; +/* Values for CheckPoint.upgradeFlag. */ +typedef enum UpgradeFlag +{ + UPGRADE_FLAG_NONE = 0, /* no in-place major version upgrade */ + UPGRADE_FLAG_BOUNDARY = 1, /* final shutdown checkpoint of the old major + * version */ + UPGRADE_FLAG_START = 2, /* first checkpoint written in the new major + * version's format */ + UPGRADE_FLAG_GOLIVE = 3, /* checkpoint that ends the migration */ +} UpgradeFlag; + /* XLOG info values for XLOG rmgr */ #define XLOG_CHECKPOINT_SHUTDOWN 0x00 #define XLOG_CHECKPOINT_ONLINE 0x10 diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index f46427258e3..9a178c05465 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6867,6 +6867,19 @@ { oid => '2848', descr => 'switch to new wal file', proname => 'pg_switch_wal', provolatile => 'v', prorettype => 'pg_lsn', proargtypes => '', prosrc => 'pg_switch_wal', proacl => '{POSTGRES=X}' }, +{ oid => '9819', + descr => 'request an upgrade boundary at the next shutdown checkpoint', + proname => 'pg_request_upgrade_boundary', provolatile => 'v', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_request_upgrade_boundary', proacl => '{POSTGRES=X}' }, +{ oid => '9822', descr => 'cancel a requested upgrade boundary', + proname => 'pg_cancel_upgrade_boundary', provolatile => 'v', + prorettype => 'void', proargtypes => '', + prosrc => 'pg_cancel_upgrade_boundary', proacl => '{POSTGRES=X}' }, +{ oid => '9820', descr => 'true if an upgrade boundary is active', + proname => 'pg_upgrade_boundary_requested', provolatile => 'v', + prorettype => 'bool', proargtypes => '', + prosrc => 'pg_upgrade_boundary_requested' }, { oid => '6305', descr => 'log details of the current snapshot to WAL', proname => 'pg_log_standby_snapshot', provolatile => 'v', prorettype => 'pg_lsn', proargtypes => '', diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 72113c5ac6e..28bcca86345 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -65,6 +65,7 @@ tests += { 't/054_unlogged_sequence_promotion.pl', 't/055_cascade_reconnect.pl', 't/056_standby_snapshot_export.pl', + 't/057_upgrade_boundary.pl', ], }, } diff --git a/src/test/recovery/t/057_upgrade_boundary.pl b/src/test/recovery/t/057_upgrade_boundary.pl new file mode 100644 index 00000000000..4e40e3bb3f1 --- /dev/null +++ b/src/test/recovery/t/057_upgrade_boundary.pl @@ -0,0 +1,105 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test marking a shutdown checkpoint as an upgrade boundary: the +# deliberate end of this major version's WAL, pending an in-place upgrade. +# The marking is visible in the control file and in the WAL itself, and a +# cluster simply restarted on the same binary cancels it. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +# Fetch one field from the control file +sub control_field +{ + my ($node, $field) = @_; + + my ($stdout, $stderr) = + run_command([ 'pg_controldata', $node->data_dir ]); + die "$field not found in control file\n" + unless $stdout =~ /^\Q$field\E:\s*(.*)$/m; + + return $1; +} + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init; +$primary->start; + +# Request state round trip on the primary +is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', 'initially not active'); +$primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 't', 'active after request'); +$primary->safe_psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); +is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', 'cancelled again'); +$primary->safe_psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); +is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', 'cancelling with nothing requested is a no-op'); + +# An online checkpoint neither consumes nor acts on a pending request; +# only a shutdown checkpoint does. +$primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +$primary->safe_psql('postgres', 'CHECKPOINT'); +is(control_field($primary, "Latest checkpoint's upgrade flag"), + 'none', 'online checkpoint is not marked as a boundary'); +is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 't', 'request stays active across an online checkpoint'); +$primary->safe_psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); + +# Shutting down with a request pending marks the shutdown checkpoint, and +# the control file keeps the marking, so the next binary started on this +# data directory can see it. +$primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +$primary->stop('fast'); +is(control_field($primary, "Latest checkpoint's upgrade flag"), + 'boundary', 'shutdown checkpoint is marked as a boundary'); + +# The flag is in the WAL itself, not just the control file. +command_like( + [ + 'pg_waldump', '-p', $primary->data_dir . '/pg_wal', + '-s', control_field($primary, 'Latest checkpoint location'), + '-n', '1' + ], + qr/CHECKPOINT_SHUTDOWN .*upgrade flag boundary/, + 'pg_waldump shows the flag on the boundary checkpoint record'); + +# pg_resetwal discards the WAL after the boundary checkpoint and writes a +# new one in its place, so the marking must not carry over to it. +command_ok([ 'pg_resetwal', $primary->data_dir ], 'pg_resetwal succeeds'); +is(control_field($primary, "Latest checkpoint's upgrade flag"), + 'none', 'pg_resetwal clears the boundary marking'); +command_like( + [ + 'pg_waldump', '-p', $primary->data_dir . '/pg_wal', + '-s', control_field($primary, 'Latest checkpoint location'), + '-n', '1' + ], + qr/CHECKPOINT_SHUTDOWN (?!.*upgrade flag)/, + 'checkpoint written by pg_resetwal is not marked'); + +# A boundary can be requested again on the reset cluster. +$primary->start; +$primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +$primary->stop('fast'); +is(control_field($primary, "Latest checkpoint's upgrade flag"), + 'boundary', 'boundary re-established after pg_resetwal'); + +# Abandon the upgrade: starting the primary on the same binary cancels the +# boundary as a side effect of writing new WAL on this timeline. +$primary->start; +ok( $primary->log_contains( + 'database system was shut down at an upgrade boundary'), + 'primary logged the cancelled boundary'); +is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', 'restarting does not re-request'); +$primary->safe_psql('postgres', 'CHECKPOINT'); +is(control_field($primary, "Latest checkpoint's upgrade flag"), + 'none', 'cancelled boundary is gone from the primary control file'); + +done_testing(); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 5d432074c2c..95ccfec4a8e 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -3320,6 +3320,7 @@ UnresolvedTup UnresolvedTupData UpdateContext UpdateStmt +UpgradeFlag UpgradeTask UpgradeTaskProcessCB UpgradeTaskReport -- 2.55.0
From 428deadc10509348737aede72643c306320d0183 Mon Sep 17 00:00:00 2001 From: John Naylor <[email protected]> Date: Wed, 26 Aug 2026 17:42:22 +0700 Subject: [PATCH v20261003 2/2] Pause archive recovery at an upgrade boundary When replay of a shutdown checkpoint marked as an upgrade boundary completes, the startup process shuts down the walreceiver so nothing past the boundary accumulates in pg_wal, and pauses to avoid fetching and reading any further WAL. Hot standby continues to serve queries while paused. Unlike resuming from a pause at a recovery target, resuming from an upgrade boundary simply continues replay. This is needed if an upgrade is abandoned, so that a standby can follow the restarted primary instead of diverging onto a new timeline, as promotion would. Shutting down a paused standby writes a restartpoint at the boundary checkpoint, so its control file reads "shut down in recovery" with the boundary marker. The new binary started on the data directory will look for those states. Restarting on the same binary starts recovery from the boundary checkpoint itself, so it re-pauses before fetching any WAL rather than falling into the retry loop. Cascaded standbys replay the same checkpoint record and pause on their own. The boundary marker must only ever mean "parked here". After a resume, replay continues past the boundary checkpoint, but the control file keeps the boundary checkpoint and its marker until replay reaches the next checkpoint record. A shutdown in that window would leave the same control file state as a parked standby, although the data is already past the boundary. Restartpoints therefore clear the marker once replay has moved beyond the checkpoint, and a restart in the same window compares the minimum recovery point against the checkpoint before re-pausing, since a crash leaves the stale marker in place. If hot standby is not active, nobody could ever resume the pause, so recovery instead shuts down cleanly, following recovery_target_action=shutdown. Crash recovery ignores boundaries entirely. A new function pg_upgrade_boundary_lsn() returns the location while paused at a boundary, letting clients distinguish a standby paused at a boundary from one paused manually. WIP: the postmaster and pg_ctl exit-status-3 messages still say "recovery target". --- doc/src/sgml/func/func-admin.sgml | 29 ++- src/backend/access/transam/xlog.c | 22 ++ src/backend/access/transam/xlogfuncs.c | 15 ++ src/backend/access/transam/xlogrecovery.c | 148 ++++++++++++ src/backend/postmaster/postmaster.c | 1 + src/bin/pg_ctl/pg_ctl.c | 1 + src/include/access/xlogrecovery.h | 9 + src/include/catalog/pg_proc.dat | 4 + src/test/recovery/t/057_upgrade_boundary.pl | 253 +++++++++++++++++++- 9 files changed, 467 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 5fb55b1e14f..0ef57042bfd 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -896,7 +896,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset linkend="functions-upgrade-boundary-table"/> control the marking of an <firstterm>upgrade boundary</firstterm>: a shutdown checkpoint that is the deliberate end of the current major version's write-ahead log, - written in preparation for an in-place major version upgrade. + written in preparation for an in-place major version upgrade. When a + standby server replays an upgrade boundary, it stops streaming and + pauses recovery, since it cannot read WAL written by a later major version. + The paused standby can then be shut down to have its binaries upgraded, + or promoted, keeping the cluster on the old major version. </para> <table id="functions-upgrade-boundary-table"> @@ -968,6 +972,23 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>. </para></entry> </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>pg_upgrade_boundary_lsn</primary> + </indexterm> + <function>pg_upgrade_boundary_lsn</function> () + <returnvalue>pg_lsn</returnvalue> + </para> + <para> + Returns the location of the upgrade boundary checkpoint record that + recovery is paused at, or <literal>NULL</literal> if it is not + paused at one. This distinguishes a standby paused at an upgrade + boundary from one paused with + <function>pg_wal_replay_pause</function>. + </para></entry> + </row> </tbody> </tgroup> </table> @@ -975,7 +996,11 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <para> If a server that was shut down at an upgrade boundary is simply started again with the same major version, the boundary is cancelled: writing - new write-ahead log on the same timeline supersedes it. + new write-ahead log on the same timeline supersedes it, and standby + servers paused at the boundary can be resumed with + <function>pg_wal_replay_resume</function>. Unlike resuming from a pause + at a recovery target, resuming from an upgrade boundary does not promote + the standby; it simply continues replay. </para> </sect2> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 83bba8220c5..dc60c55b1ac 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -8520,6 +8520,25 @@ RecoveryRestartPoint(const CheckPoint *checkPoint, XLogReaderState *record) SpinLockRelease(&XLogCtl->info_lck); } +/* + * A boundary marking in the control file says this node is parked at that + * checkpoint, which is what a new binary started on this data directory acts + * on. Once replay has moved past it, as after an abandoned upgrade, the + * marking no longer describes this node, even though that checkpoint stays + * the restartpoint until the next checkpoint record is replayed. + * + * Replay is past the checkpoint once the last replayed record starts after + * it. Caller must hold ControlFileLock exclusively and write the control + * file. + */ +static void +ClearSupersededUpgradeBoundary(void) +{ + if (ControlFile->checkPointCopy.upgradeFlag == UPGRADE_FLAG_BOUNDARY && + GetXLogReplayReadRecPtr() > ControlFile->checkPoint) + ControlFile->checkPointCopy.upgradeFlag = UPGRADE_FLAG_NONE; +} + /* * Establish a restartpoint if possible. * @@ -8628,6 +8647,7 @@ CreateRestartPoint(int flags) LWLockAcquire(ControlFileLock, LW_EXCLUSIVE); ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY; + ClearSupersededUpgradeBoundary(); if (catchUpChecksums) { ControlFile->data_checksum_version = checksum_state; @@ -8743,6 +8763,8 @@ CreateRestartPoint(int flags) ControlFile->state = DB_SHUTDOWNED_IN_RECOVERY; } + ClearSupersededUpgradeBoundary(); + /* * Persist the data checksum state of this node. Not the state of the * replayed checkpoint: that one belongs to the node that wrote it and diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index bffd724b473..b0099e2e55d 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -280,6 +280,21 @@ pg_upgrade_boundary_requested(PG_FUNCTION_ARGS) PG_RETURN_BOOL(UpgradeBoundaryActive()); } +/* + * pg_upgrade_boundary_lsn: location of the upgrade boundary if recovery + * is paused at one, or NULL otherwise. + */ +Datum +pg_upgrade_boundary_lsn(PG_FUNCTION_ARGS) +{ + XLogRecPtr lsn = GetUpgradeBoundaryLSN(); + + if (XLogRecPtrIsInvalid(lsn)) + PG_RETURN_NULL(); + + PG_RETURN_LSN(lsn); +} + /* * pg_log_standby_snapshot: call LogStandbySnapshot() * diff --git a/src/backend/access/transam/xlogrecovery.c b/src/backend/access/transam/xlogrecovery.c index fff8d57ac61..ae2783bb2e9 100644 --- a/src/backend/access/transam/xlogrecovery.c +++ b/src/backend/access/transam/xlogrecovery.c @@ -174,6 +174,9 @@ static TimeLineID CheckPointTLI = 0; static XLogRecPtr RedoStartLSN = InvalidXLogRecPtr; static TimeLineID RedoStartTLI = 0; +/* Is the checkpoint we start recovery from an upgrade boundary? */ +static bool CheckPointIsUpgradeBoundary = false; + /* * Local copy of SharedHotStandbyActive variable. False actually means "not * known, need to check the shared state". @@ -361,6 +364,8 @@ static void verifyBackupPageConsistency(XLogReaderState *record); static bool recoveryStopsBefore(XLogReaderState *record); static bool recoveryStopsAfter(XLogReaderState *record); +static void UpgradeBoundaryReached(XLogRecPtr lsn); +static void CheckForUpgradeBoundary(XLogReaderState *record); static char *getRecoveryStopReason(void); static void recoveryPausesHere(bool endOfRecovery); static bool recoveryApplyDelay(XLogReaderState *record); @@ -979,6 +984,8 @@ InitWalRecovery(ControlFileData *ControlFile, bool *wasShutdown_ptr, abortedRecPtr = InvalidXLogRecPtr; missingContrecPtr = InvalidXLogRecPtr; + CheckPointIsUpgradeBoundary = (checkPoint.upgradeFlag == UPGRADE_FLAG_BOUNDARY); + *wasShutdown_ptr = wasShutdown; *haveBackupLabel_ptr = haveBackupLabel; *haveTblspcMap_ptr = haveTblspcMap; @@ -1669,6 +1676,20 @@ PerformWalRecovery(void) */ CheckRecoveryConsistency(); + /* + * If recovery starts from a checkpoint that is an upgrade boundary -- as + * on a standby that was restarted after pausing at one -- the boundary is + * already reached: pause again before fetching any WAL. + * + * That is not the case if replay had already resumed past the boundary + * before the shutdown, as after an abandoned upgrade: the boundary stays + * the last restartpoint until the next checkpoint record is replayed, but + * the minimum recovery point has moved on. + */ + if (CheckPointIsUpgradeBoundary && + minRecoveryPoint <= XLogRecoveryCtl->lastReplayedEndRecPtr) + UpgradeBoundaryReached(CheckPointLoc); + /* * Find the first record that logically follows the checkpoint --- it * might physically precede it, though. @@ -1808,6 +1829,14 @@ PerformWalRecovery(void) WaitLSNWakeup(WAIT_LSN_TYPE_STANDBY_FLUSH, XLogRecoveryCtl->lastReplayedEndRecPtr); + /* + * If the record just applied marked an upgrade boundary, stop + * fetching WAL and pause here. This must happen before we try to + * read the next record, since this server version may not be able + * read it. + */ + CheckForUpgradeBoundary(xlogreader); + /* Exit loop if we reached inclusive recovery target */ if (recoveryStopsAfter(xlogreader)) { @@ -2905,6 +2934,93 @@ getRecoveryStopReason(void) return pstrdup(reason); } +/* + * Check whether the record just replayed was a shutdown checkpoint marked + * as an upgrade boundary, and if so, stop streaming and pause recovery. + * + * An upgrade boundary is the end of the WAL of the binary that wrote it. + * Subsequent WAL is normally written by binary of a later major version, + * which we presume this server cannot read. + * + * Pausing allows the operator to act appropriately: + * - shut down cleanly to swap in the new binaries + * - resume replay if the upgrade was abandoned + * - promote to fail over, keeping this major version + * + * Unlike at a recovery target, resuming does not promote. + * + * If hot standby is not active, nobody can resume us, so shut down + * cleanly instead of pausing, as with a recovery target with + * recovery_target_action=shutdown. + */ +static void +CheckForUpgradeBoundary(XLogReaderState *record) +{ + CheckPoint checkPoint; + + if (XLogRecGetRmid(record) != RM_XLOG_ID || + (XLogRecGetInfo(record) & ~XLR_INFO_MASK) != XLOG_CHECKPOINT_SHUTDOWN) + return; + + memcpy(&checkPoint, XLogRecGetData(record), sizeof(CheckPoint)); + if (checkPoint.upgradeFlag != UPGRADE_FLAG_BOUNDARY) + return; + + UpgradeBoundaryReached(record->ReadRecPtr); +} + +/* + * Act on a reached upgrade boundary: stop streaming and pause, or shut + * down cleanly if hot standby is not active. Called either when replay + * of a boundary checkpoint record completes, or at the start of recovery + * when the starting checkpoint itself is a boundary. + */ +static void +UpgradeBoundaryReached(XLogRecPtr lsn) +{ + /* Upgrade boundaries are not relevant to plain crash recovery */ + if (!ArchiveRecoveryRequested) + return; + + /* Make the boundary visible to pg_upgrade_boundary_lsn() */ + SpinLockAcquire(&XLogRecoveryCtl->info_lck); + XLogRecoveryCtl->upgradeBoundaryLSN = lsn; + SpinLockRelease(&XLogRecoveryCtl->info_lck); + + ereport(LOG, + (errmsg("recovery has reached an upgrade boundary at %X/%08X", + LSN_FORMAT_ARGS(lsn)))); + + /* + * Stop streaming, so that nothing past the boundary accumulates in + * pg_wal. If replay is later resumed, streaming is restarted on demand. + */ + XLogShutdownWalRcv(); + + if (!LocalHotStandbyActive) + { + ereport(LOG, + (errmsg("shutting down at upgrade boundary"))); + + /* + * exit with special return code to request shutdown of postmaster. + * Log messages issued from postmaster. + */ + proc_exit(3); + } + + SetRecoveryPause(true); + recoveryPausesHere(false); + + /* + * Replay was resumed or promotion requested: either way, recovery is no + * longer paused at the boundary. + */ + SpinLockAcquire(&XLogRecoveryCtl->info_lck); + XLogRecoveryCtl->upgradeBoundaryLSN = InvalidXLogRecPtr; + SpinLockRelease(&XLogRecoveryCtl->info_lck); +} + /* * Wait until shared recoveryPauseState is set to RECOVERY_NOT_PAUSED. * @@ -3093,6 +3209,22 @@ SetRecoveryPause(bool recoveryPause) ConditionVariableBroadcast(&XLogRecoveryCtl->recoveryNotPausedCV); } +/* + * Get the location of the upgrade boundary if recovery is paused at one, or + * InvalidXLogRecPtr otherwise. + */ +XLogRecPtr +GetUpgradeBoundaryLSN(void) +{ + XLogRecPtr lsn; + + SpinLockAcquire(&XLogRecoveryCtl->info_lck); + lsn = XLogRecoveryCtl->upgradeBoundaryLSN; + SpinLockRelease(&XLogRecoveryCtl->info_lck); + + return lsn; +} + /* * Confirm the recovery pause by setting the recovery pause state to * RECOVERY_PAUSED. @@ -4581,6 +4713,22 @@ HotStandbyActiveInReplay(void) return LocalHotStandbyActive; } +/* + * Get the start of the last record replayed, or InvalidXLogRecPtr if + * recovery started at a checkpoint's redo point rather than at its record. + */ +XLogRecPtr +GetXLogReplayReadRecPtr(void) +{ + XLogRecPtr recptr; + + SpinLockAcquire(&XLogRecoveryCtl->info_lck); + recptr = XLogRecoveryCtl->lastReplayedReadRecPtr; + SpinLockRelease(&XLogRecoveryCtl->info_lck); + + return recptr; +} + /* * Get latest redo apply position. * diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c index ef300a6c45a..4510caca90f 100644 --- a/src/backend/postmaster/postmaster.c +++ b/src/backend/postmaster/postmaster.c @@ -2294,6 +2294,7 @@ process_pm_child_exit(void) if (EXIT_STATUS_3(exitstatus)) { + /* WIP: this is misleading if shutdown at a version boundary */ ereport(LOG, (errmsg("shutdown at recovery target"))); StartupStatus = STARTUP_NOT_RUNNING; diff --git a/src/bin/pg_ctl/pg_ctl.c b/src/bin/pg_ctl/pg_ctl.c index 199f6c55b4a..2a7244df907 100644 --- a/src/bin/pg_ctl/pg_ctl.c +++ b/src/bin/pg_ctl/pg_ctl.c @@ -1007,6 +1007,7 @@ do_start(void) exit(1); break; case POSTMASTER_SHUTDOWN_IN_RECOVERY: + /* WIP: need separate message for upgrade boundary? */ print_msg(_(" done\n")); print_msg(_("server shut down because of recovery target settings\n")); break; diff --git a/src/include/access/xlogrecovery.h b/src/include/access/xlogrecovery.h index 8786b6d3a8e..a13f4c34ef1 100644 --- a/src/include/access/xlogrecovery.h +++ b/src/include/access/xlogrecovery.h @@ -126,6 +126,13 @@ typedef struct XLogRecoveryCtlData RecoveryPauseState recoveryPauseState; ConditionVariable recoveryNotPausedCV; + /* + * Location of the upgrade boundary checkpoint record that recovery is + * paused at, or InvalidXLogRecPtr if it is not paused at one. Protected + * by info_lck. + */ + XLogRecPtr upgradeBoundaryLSN; + slock_t info_lck; /* locks shared variables shown above */ } XLogRecoveryCtlData; @@ -217,8 +224,10 @@ extern void RemovePromoteSignalFiles(void); extern bool HotStandbyActive(void); extern XLogRecPtr GetXLogReplayRecPtr(TimeLineID *replayTLI); +extern XLogRecPtr GetXLogReplayReadRecPtr(void); extern RecoveryPauseState GetRecoveryPauseState(void); extern void SetRecoveryPause(bool recoveryPause); +extern XLogRecPtr GetUpgradeBoundaryLSN(void); extern void GetXLogReceiptTime(TimestampTz *rtime, bool *fromStream); extern TimestampTz GetLatestXTime(void); extern TimestampTz GetCurrentChunkReplayStartTime(void); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 9a178c05465..c6c95251ecf 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -6880,6 +6880,10 @@ proname => 'pg_upgrade_boundary_requested', provolatile => 'v', prorettype => 'bool', proargtypes => '', prosrc => 'pg_upgrade_boundary_requested' }, +{ oid => '9821', descr => 'upgrade boundary location recovery is paused at', + proname => 'pg_upgrade_boundary_lsn', provolatile => 'v', + prorettype => 'pg_lsn', proargtypes => '', + prosrc => 'pg_upgrade_boundary_lsn' }, { oid => '6305', descr => 'log details of the current snapshot to WAL', proname => 'pg_log_standby_snapshot', provolatile => 'v', prorettype => 'pg_lsn', proargtypes => '', diff --git a/src/test/recovery/t/057_upgrade_boundary.pl b/src/test/recovery/t/057_upgrade_boundary.pl index 4e40e3bb3f1..12ede030713 100644 --- a/src/test/recovery/t/057_upgrade_boundary.pl +++ b/src/test/recovery/t/057_upgrade_boundary.pl @@ -4,12 +4,17 @@ # deliberate end of this major version's WAL, pending an in-place upgrade. # The marking is visible in the control file and in the WAL itself, and a # cluster simply restarted on the same binary cancels it. +# +# A standby replaying a boundary stops streaming and pauses recovery; from +# there it can be shut down (to swap binaries), resumed (if the upgrade was +# abandoned and the primary keeps going), or promoted. use strict; use warnings FATAL => 'all'; use PostgreSQL::Test::Cluster; use PostgreSQL::Test::Utils; use Test::More; +use Time::HiRes qw(usleep); # Fetch one field from the control file sub control_field @@ -25,21 +30,29 @@ sub control_field } my $primary = PostgreSQL::Test::Cluster->new('primary'); -$primary->init; +$primary->init(allows_streaming => 1); $primary->start; # Request state round trip on the primary -is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), - 'f', 'initially not active'); +is( $primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', + 'initially not active'); $primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); -is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), - 't', 'active after request'); +is( $primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 't', + 'active after request'); $primary->safe_psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); -is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), - 'f', 'cancelled again'); +is( $primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', + 'cancelled again'); $primary->safe_psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); -is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), - 'f', 'cancelling with nothing requested is a no-op'); +is( $primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', + 'cancelling with nothing requested is a no-op'); +is( $primary->safe_psql( + 'postgres', 'SELECT pg_upgrade_boundary_lsn() IS NULL'), + 't', + 'no boundary replayed on the primary'); # An online checkpoint neither consumes nor acts on a pending request; # only a shutdown checkpoint does. @@ -47,8 +60,9 @@ $primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); $primary->safe_psql('postgres', 'CHECKPOINT'); is(control_field($primary, "Latest checkpoint's upgrade flag"), 'none', 'online checkpoint is not marked as a boundary'); -is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), - 't', 'request stays active across an online checkpoint'); +is( $primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 't', + 'request stays active across an online checkpoint'); $primary->safe_psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); # Shutting down with a request pending marks the shutdown checkpoint, and @@ -96,10 +110,223 @@ $primary->start; ok( $primary->log_contains( 'database system was shut down at an upgrade boundary'), 'primary logged the cancelled boundary'); -is($primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), - 'f', 'restarting does not re-request'); +is( $primary->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', + 'restarting does not re-request'); + +# No checkpoint has been written since, so a crash now makes recovery start +# at the boundary checkpoint record itself. Crash recovery ignores the +# marking: it neither pauses nor shuts down, it just comes up. +$primary->safe_psql('postgres', + 'CREATE TABLE crash (a int); INSERT INTO crash VALUES (1)'); +$primary->stop('immediate'); +$primary->start; +ok($primary->log_contains('automatic recovery in progress'), + 'primary went through crash recovery'); +ok(!$primary->log_contains('reached an upgrade boundary'), + 'crash recovery ignored the boundary'); +is($primary->safe_psql('postgres', 'SELECT count(*) FROM crash'), + '1', 'crash recovery replayed past the boundary'); + $primary->safe_psql('postgres', 'CHECKPOINT'); is(control_field($primary, "Latest checkpoint's upgrade flag"), 'none', 'cancelled boundary is gone from the primary control file'); +# Everything above needs only a primary. Add a standby and walk it through +# a boundary: it must be streaming when the primary shuts down, since the +# boundary reaches it as WAL like anything else. +$primary->backup('bkp'); +my $standby = PostgreSQL::Test::Cluster->new('standby'); +$standby->init_from_backup($primary, 'bkp', has_streaming => 1); +$standby->start; + +# The request and cancel functions are refused during recovery +my ($ret, $stdout, $stderr) = + $standby->psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +like( + $stderr, + qr/recovery is in progress/, + 'requesting is refused on a standby'); +($ret, $stdout, $stderr) = + $standby->psql('postgres', 'SELECT pg_cancel_upgrade_boundary()'); +like( + $stderr, + qr/recovery is in progress/, + 'cancelling is refused on a standby'); +is( $standby->safe_psql('postgres', 'SELECT pg_upgrade_boundary_requested()'), + 'f', + 'request state reads as false on a standby'); +is( $standby->safe_psql( + 'postgres', 'SELECT pg_upgrade_boundary_lsn() IS NULL'), + 't', + 'no boundary replayed yet on the standby'); + +$primary->safe_psql('postgres', + 'CREATE TABLE t (a int); INSERT INTO t VALUES (1);'); +$primary->wait_for_catchup($standby); + +# Request a boundary and shut the primary down: the standby replays the +# marked checkpoint and pauses there. +$primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +$primary->stop('fast'); +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for standby to pause at upgrade boundary"; +ok(1, 'standby paused after replaying the upgrade boundary'); + +my $boundary_lsn = + $standby->safe_psql('postgres', 'SELECT pg_upgrade_boundary_lsn()'); +isnt($boundary_lsn, '', 'pg_upgrade_boundary_lsn reports the boundary'); +is( $standby->safe_psql( + 'postgres', 'SELECT count(*) FROM pg_stat_wal_receiver'), + '0', + 'walreceiver was stopped at the boundary'); +is($standby->safe_psql('postgres', 'SELECT count(*) FROM t'), + '1', 'paused standby still serves reads'); +ok($standby->log_contains('recovery has reached an upgrade boundary'), + 'standby logged the boundary'); + +# Abandon this upgrade too: restart the primary and resume the standby. +# Resuming continues replay; unlike a recovery target, it does not promote. +$primary->start; +$primary->safe_psql('postgres', 'INSERT INTO t VALUES (2)'); +$standby->safe_psql('postgres', 'SELECT pg_wal_replay_resume()'); +$primary->wait_for_catchup($standby); +is($standby->safe_psql('postgres', 'SELECT count(*) FROM t'), + '2', 'standby resumed following the primary after abandoned upgrade'); +is($standby->safe_psql('postgres', 'SELECT pg_is_in_recovery()'), + 't', 'resuming from a boundary did not promote'); +is( $standby->safe_psql( + 'postgres', 'SELECT pg_upgrade_boundary_lsn() IS NULL'), + 't', + 'boundary location is cleared once replay resumes'); + +# A later manual pause is not a boundary pause. +$standby->safe_psql('postgres', 'SELECT pg_wal_replay_pause()'); +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for standby to pause manually"; +is( $standby->safe_psql( + 'postgres', 'SELECT pg_upgrade_boundary_lsn() IS NULL'), + 't', + 'manual pause after a boundary is not reported as a boundary'); +$standby->safe_psql('postgres', 'SELECT pg_wal_replay_resume()'); + +# The resumed standby still has the boundary checkpoint as its restartpoint +# until it replays the primary's next checkpoint. Shutting down in that +# window must not pin the boundary in the control file, since the data is +# already past it, and restarting must not treat the superseded boundary as +# reached: the standby must start up and keep following. +$standby->stop('fast'); +is(control_field($standby, "Latest checkpoint's upgrade flag"), + 'none', 'control file does not pin a boundary replay has moved past'); +$standby->start; +is( $standby->safe_psql('postgres', 'SELECT pg_get_wal_replay_pause_state()'), + 'not paused', + 'standby restarted past a boundary does not pause'); +$primary->safe_psql('postgres', 'INSERT INTO t VALUES (3)'); +$primary->wait_for_catchup($standby); +is($standby->safe_psql('postgres', 'SELECT count(*) FROM t'), + '3', + 'restarted standby follows the primary past the superseded boundary'); + +# Add a cascading standby fed by the first one. It sees the boundary as +# the same checkpoint record, relayed by a standby that pauses on it, so it +# needs no coordination to pause itself. +my $cascade = PostgreSQL::Test::Cluster->new('cascade'); +$standby->backup('bkp2'); +$cascade->init_from_backup($standby, 'bkp2', has_streaming => 1); +$cascade->start; + +# Request again and shut down: the standby pauses at a second, distinct +# boundary. Make sure both are streaming first, so the boundary reaches +# them. +$primary->poll_query_until('postgres', + "SELECT count(*) = 1 FROM pg_stat_replication WHERE state = 'streaming'") + or die "timed out waiting for standby to stream"; +$standby->poll_query_until('postgres', + "SELECT count(*) = 1 FROM pg_stat_replication WHERE state = 'streaming'") + or die "timed out waiting for cascading standby to stream"; +$primary->safe_psql('postgres', 'SELECT pg_request_upgrade_boundary()'); +$primary->stop('fast'); +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for standby to pause at second upgrade boundary"; +my $boundary_lsn2 = + $standby->safe_psql('postgres', 'SELECT pg_upgrade_boundary_lsn()'); +isnt($boundary_lsn2, $boundary_lsn, 'second boundary has a new location'); + +$cascade->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for cascading standby to pause"; +is($cascade->safe_psql('postgres', 'SELECT pg_upgrade_boundary_lsn()'), + $boundary_lsn2, 'cascading standby paused at the same boundary'); +is( $cascade->safe_psql( + 'postgres', 'SELECT count(*) FROM pg_stat_wal_receiver'), + '0', + 'cascading standby stopped its walreceiver'); +$cascade->stop('fast'); +is(control_field($cascade, "Latest checkpoint's upgrade flag"), + 'boundary', 'cascading standby control file records the boundary'); + +# Shutting the paused standby down pins the boundary in its own control +# file, where the next binary started on that data directory can see it. +$standby->stop('fast'); +is( control_field($standby, 'Database cluster state'), + 'shut down in recovery', + 'paused standby shut down cleanly'); +is(control_field($standby, "Latest checkpoint's upgrade flag"), + 'boundary', 'standby control file records the boundary'); + +# A restarted standby starts recovery from the boundary checkpoint itself +# and re-pauses before fetching any WAL. +$standby->start; +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for restarted standby to re-pause"; +is($standby->safe_psql('postgres', 'SELECT pg_upgrade_boundary_lsn()'), + $boundary_lsn2, 'restarted standby re-paused at the same boundary'); + +# Without hot standby nobody could resume a pause, so recovery shuts down +# cleanly at the boundary instead. The node exits on its own, so drive +# pg_ctl directly rather than through start(), which expects it to stay up. +$standby->stop('fast'); +$standby->append_conf('postgresql.conf', 'hot_standby = off'); +command_ok( + [ + 'pg_ctl', + '--pgdata' => $standby->data_dir, + '--log' => $standby->logfile, + 'start' + ], + 'pg_ctl start returns with hot standby off'); +my $attempts = 10 * $PostgreSQL::Test::Utils::timeout_default; +while (-f $standby->data_dir . '/postmaster.pid' && $attempts-- > 0) +{ + usleep(100_000); +} +ok(!-f $standby->data_dir . '/postmaster.pid', + 'standby without hot standby shut down at the boundary'); +ok($standby->log_contains('shutting down at upgrade boundary'), + 'standby logged the shutdown'); +is( control_field($standby, 'Database cluster state'), + 'shut down in recovery', + 'shutdown at the boundary was clean'); +is(control_field($standby, "Latest checkpoint's upgrade flag"), + 'boundary', 'control file still records the boundary'); + +# Back to hot standby for the last exit. +$standby->append_conf('postgresql.conf', 'hot_standby = on'); +$standby->start; +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for standby to re-pause with hot standby on"; + +# Promotion is the other valid exit: fail over onto the old major version. +$standby->promote; +$standby->poll_query_until('postgres', 'SELECT NOT pg_is_in_recovery()') + or die "timed out waiting for standby promotion"; +is($standby->safe_psql('postgres', 'SELECT count(*) FROM t'), + '3', 'promoted standby has all data up to the boundary'); + done_testing(); -- 2.55.0
