On Thu, 22 Oct 2026 at 21:43, Andrey Borodin <[email protected]> wrote: > > Hi hackers, > > A standby can fall behind the WAL retained by its upstream while the > missing segments are still available in the archive. We would like to > keep that history in the archive, not retain all of it locally for a > replication slot. Giving the receiver its own restore_command is not > always an option either: we may be able to grant replication access > without giving it access to the archive. Could the sender provide the > missing WAL over the existing replication connection instead? > > Fujii proposed this in 2010 [0], and Japin posted a patch in 2025 [1]. > I would like to revisit it with the archive access restriction above as > the use case, and with private copies instead of restoring into the > normal WAL namespace. > > Attached is a prototype. With wal_sender_restore = on, a physical sender > that cannot open a segment in pg_wal tries restore_command before > failing. This works on a primary or a [cascading] standby. > > The temporary storage and cleanup questions came up when discussing > reviving lost replication slots [2]. Here each sender keeps at most one > restored segment in its own pg_wal/xlogtemp.*.walsender directory. > > Alexander pointed out the cost of repeated failed restores on the > primary [3]. A request arriving within one second of a failed restore > waits for the remainder of that interval. > > The existing timeline and flush/replay limits still apply. This does not > enable logical decoding from the archive or revive an invalidated slot. > It can serve a receiver without a slot, or with a usable physical slot. > Joao's proposal to revalidate physical slots [4] addresses that separate > part of the problem. > > The prototype runs restore_command synchronously. A slow fetch prevents > keepalives and feedback processing, can cause receiver timeouts, and can > delay shutdown. An interruptible implementation would need to manage the > child command while servicing the replication connection. > > What do you think? > > > Best regards, Andrey Borodin. > > [0] > https://www.postgresql.org/message-id/[email protected] > [1] > https://www.postgresql.org/message-id/me0p300mb0445c80039d6534452a87539b6...@me0p300mb0445.ausp300.prod.outlook.com > [2] > https://www.postgresql.org/message-id/CALj2ACVUgCYVgKc42GDCFZo_=xjyydthxbhzyht3s4uqazl...@mail.gmail.com > [3] > https://www.postgresql.org/message-id/CAFh8B=nH=41scx3f_eah2h5o1w-nj8b7ucmp_0z4p4wsv2t...@mail.gmail.com > [4] https://commitfest.postgresql.org/patch/6865/
Hi! Reviewed patch. There are issues with test races, and consequent promotions , but they are already fixed in [0]. I briefly looked at them - should be OK. Will wait for list posting. There are additional improvements possible. 1. TIMELINE_HISTORY support. If the replica contains only the current timeline history file, say X, the replication proto client can request TIMELINE_HISTORY X - 1 and fail. attached 0001 implements that. This is a separate improvement though and I do not intend to merge this in one commit with the main feature. 2.WALbuffers WAL read can fail due to on-disk xlog files removal. Being discovered while reviewing this, this is a separate issue, so I reported it as [1]. === Some nits, in unsorted level of severity Also, this magic number > /* Minimum delay after a failed archive restore, in milliseconds. */ > #define WALSND_RESTORE_DELAY 1000 Should we make this a GUC? > /* Time of last archive restore failure, protected by restore_mutex. */ > slock_t restore_mutex; > TimestampTz lastRestoreFailure; Can we use a single atomic integer? [0] https://github.com/x4m/postgres_g/commits/refs/heads/walsender-archive-v1-followups/ [1] https://www.postgresql.org/message-id/CALdSSPid5M%3DqE9UHz5duQE_%2BxJF0bn1OQE4W8fxb3%2BBtZ0Yk7w%40mail.gmail.com -- Best regards, Kirill Reshke
From bb8b07a28d3c4f8667c7a1d65ff4cce1fad42072 Mon Sep 17 00:00:00 2001 Message-ID: <bb8b07a28d3c4f8667c7a1d65ff4cce1fad42072.1792838665.git.localhost> In-Reply-To: <cover.1792838665.git.localhost> References: <[email protected]> <cover.1792838665.git.localhost> From: root <localhost> Date: Sat, 24 Oct 2026 15:27:24 +0500 Subject: [PATCH 1/4] Allow TIMELINE_HISTORY to restore history from the archive TIMELINE_HISTORY currently requires the requested history file to exist in pg_wal. Allow the command to fetch a missing file with restore_command when wal_sender_restore is enabled. This extends archive restoration to timeline metadata on both primary and standby servers. Reuse the per-sender restore path, cleanup and failure delay used for WAL segments. Return the restored history to the client without installing it in pg_wal. Test local and archived history on physical and database replication connections, disabled fallback, missing archived files, repeated requests and cleanup. An ordinary standby initialized from a recent base backup provides a case with the current history file but no intermediate history file. Request the missing history directly with TIMELINE_HISTORY and check that upstream streaming replication continues. Discussion: [email protected]">https://www.postgresql.org/message-id/[email protected] --- doc/src/sgml/config.sgml | 19 +-- src/backend/replication/walsender.c | 55 ++++++-- src/test/recovery/meson.build | 1 + .../t/063_walsender_archive_history.pl | 121 ++++++++++++++++++ 4 files changed, 174 insertions(+), 22 deletions(-) create mode 100644 src/test/recovery/t/063_walsender_archive_history.pl diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 04160f2d53e..2b7f40d8e2b 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4328,9 +4328,9 @@ include_dir 'conf.d' <para> When <xref linkend="guc-wal-sender-restore"/> is enabled, this command - also supplies missing WAL to physical replication clients, including - outside recovery. See that setting for the requirements on the - command in this case. + also supplies missing WAL and timeline history files to physical + replication clients, including outside recovery. See that setting + for the requirements on the command in this case. </para> <para> @@ -5037,14 +5037,15 @@ HINT: If it is safe for all REPLICATION users to use this library as an output <listitem> <para> Allows physical replication senders to use - <xref linkend="guc-restore-command"/> when a requested WAL segment is - missing from <filename>pg_wal</filename>. This applies to both primary - and standby servers. The default is <literal>off</literal>. + <xref linkend="guc-restore-command"/> when a requested WAL segment or + timeline history file is missing from <filename>pg_wal</filename>. + This applies to both primary and standby servers. The default is + <literal>off</literal>. This parameter can only be set in the <filename>postgresql.conf</filename> file or on the server command line. </para> <para> - Each sender restores one segment at a time into a private temporary + Each sender restores one file at a time into a private temporary directory in <filename>pg_wal</filename>. These copies are not used by recovery or archived again, and checkpoints do not recycle them. They are removed when no longer needed or when the sender exits. @@ -5064,7 +5065,7 @@ HINT: If it is safe for all REPLICATION users to use this library as an output command. The failure timestamp is shared across senders, so this delay also applies after reconnecting. Each request attempts the restore only once, and senders can restore files concurrently. - Requests for other archived segments are also delayed, but reads + Requests for other archived files are also delayed, but reads from <filename>pg_wal</filename> and restoration by recovery are unaffected. </para> @@ -5078,7 +5079,7 @@ HINT: If it is safe for all REPLICATION users to use this library as an output <para> This does not change which timeline or WAL positions may be streamed, enable logical decoding from archived WAL, or make an invalidated - replication slot usable again. If the segment cannot be restored, + replication slot usable again. If the file cannot be restored, the replication command fails as it does when this setting is disabled. </para> </listitem> diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index f98932080ec..5ef89dbd43a 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -164,7 +164,7 @@ bool wake_wal_senders = false; */ static XLogReaderState *xlogreader = NULL; -/* Private archive copy for physical streaming, never installed as local WAL. */ +/* Private archive copy for physical replication, never installed as local WAL. */ static char restoreDir[MAXPGPATH]; static int restoredFile = -1; static bool restoreInProgress = false; @@ -336,6 +336,7 @@ static void WalSndSegmentOpen(XLogReaderState *state, XLogSegNo nextSegNo, TimeLineID *tli_p); static void WalSndSegmentClose(XLogReaderState *state); static bool WalSndRestoreSegment(XLogReaderState *state, const char *xlogfname); +static bool WalSndRestoreFile(const char *xlogfname, off_t expectedSize); static void WalSndRestoreCleanup(int code, Datum arg); @@ -640,6 +641,7 @@ SendTimeLineHistory(TimeLineHistoryCmd *cmd) off_t histfilelen; size_t bytesleft; Size len; + bool restored = false; dest = CreateDestReceiver(DestRemoteSimple); @@ -666,6 +668,19 @@ SendTimeLineHistory(TimeLineHistoryCmd *cmd) pq_sendbytes(&buf, histfname, len); fd = OpenTransientFile(path, O_RDONLY | PG_BINARY); + if (fd < 0 && errno == ENOENT && wal_sender_restore) + { + int save_errno = errno; + + /* A recent base backup need not contain intermediate history files. */ + if (WalSndRestoreFile(histfname, 0)) + { + fd = restoredFile; + restored = true; + } + else + errno = save_errno; + } if (fd < 0) ereport(ERROR, (errcode_for_file_access(), @@ -724,7 +739,9 @@ SendTimeLineHistory(TimeLineHistoryCmd *cmd) bytesleft -= nread; } - if (CloseTransientFile(fd) != 0) + if (restored) + WalSndRestoreCleanup(0, 0); + else if (CloseTransientFile(fd) != 0) ereport(ERROR, (errcode_for_file_access(), errmsg("could not close file \"%s\": %m", path))); @@ -3489,15 +3506,31 @@ WalSndSegmentClose(XLogReaderState *state) wal_segment_close(state); } +/* Restore a WAL segment and register it with the physical reader. */ +static bool +WalSndRestoreSegment(XLogReaderState *state, const char *xlogfname) +{ + if (!WalSndRestoreFile(xlogfname, wal_segment_size)) + return false; + + state->seg.ws_file = restoredFile; + XLogReaderRegisterResetCallback(state); + + INJECTION_POINT("walsender-after-archive-restore", NULL); + return true; +} + /* - * Restore one segment without exposing it to checkpoint recycling, archiving, - * recovery, or other walsenders. RemoveTempXlogFiles removes the directory - * after a crash. Using a private directory also prevents a surviving restore - * command from recreating its destination after crash cleanup, provided the - * command does not itself create the destination directory. + * Restore one WAL or history file without exposing it to checkpoint recycling, + * archiving, recovery, or other walsenders. The caller must arrange to call + * WalSndRestoreCleanup when it no longer needs the file. RemoveTempXlogFiles + * removes the directory after a crash. Using a private directory also + * prevents a surviving restore command from recreating its destination after + * crash cleanup, provided the command does not itself create the destination + * directory. */ static bool -WalSndRestoreSegment(XLogReaderState *state, const char *xlogfname) +WalSndRestoreFile(const char *xlogfname, off_t expectedSize) { char dir[MAXPGPATH]; char recovername[MAXPGPATH]; @@ -3553,7 +3586,7 @@ WalSndRestoreSegment(XLogReaderState *state, const char *xlogfname) /* WalSndErrorCleanup and the exit callback handle failures from here on. */ INJECTION_POINT("walsender-before-archive-restore", NULL); if (!RestoreArchivedFile(path, xlogfname, recovername, - wal_segment_size, false)) + expectedSize, false)) { WalSndRestoreCleanup(0, 0); return false; @@ -3564,11 +3597,7 @@ WalSndRestoreSegment(XLogReaderState *state, const char *xlogfname) ereport(ERROR, (errcode_for_file_access(), errmsg("could not open file \"%s\": %m", path))); - state->seg.ws_file = restoredFile; - XLogReaderRegisterResetCallback(state); - restoreInProgress = false; - INJECTION_POINT("walsender-after-archive-restore", NULL); return true; } diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index afc176e17f7..1f9de450ffe 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -71,6 +71,7 @@ tests += { 't/060_multixact_crashed_updater.pl', 't/061_walsender_archive.pl', 't/062_walsender_archive_timelines.pl', + 't/063_walsender_archive_history.pl', ], }, } diff --git a/src/test/recovery/t/063_walsender_archive_history.pl b/src/test/recovery/t/063_walsender_archive_history.pl new file mode 100644 index 00000000000..1334b827b2e --- /dev/null +++ b/src/test/recovery/t/063_walsender_archive_history.pl @@ -0,0 +1,121 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# A recent base backup need not contain intermediate timeline history files. +# Request them directly from an ordinary standby, without a downstream standby. +use strict; +use warnings FATAL => 'all'; + +use FindBin; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $helper = "$FindBin::RealBin/walsender_restore"; +my $scratch_count = q{SELECT count(*) FROM pg_ls_dir('pg_wal') AS f + WHERE f LIKE 'xlogtemp.%.walsender'}; +my $primary = PostgreSQL::Test::Cluster->new('history_primary'); +$primary->init(allows_streaming => 1, has_archiving => 1); +$primary->start; +$primary->safe_psql('postgres', + 'CREATE TABLE history_test (i int); INSERT INTO history_test VALUES (1)'); + +# Create timelines 2 and 3 before taking a fresh backup on timeline 3. +for (1 .. 2) +{ + $primary->stop('immediate'); + $primary->set_standby_mode; + $primary->start; + $primary->promote; +} +my $archive = $primary->archive_dir; +$primary->poll_query_until('postgres', + "SELECT (pg_stat_file('$archive/00000002.history', true)).size > 0") + or BAIL_OUT('intermediate history was not archived'); +$primary->backup('base'); + +my $standby = PostgreSQL::Test::Cluster->new('history_standby'); +$standby->init_from_backup($primary, 'base', has_streaming => 1); +my $restore_command = qq{"$^X" "$helper" "$archive" "%f" "%p" "%r"}; +$restore_command =~ s/\\/\\\\/g; +$restore_command =~ s/'/''/g; +$standby->append_conf('postgresql.conf', qq( +archive_mode = off +restore_command = '$restore_command' +)); +$standby->start; +$primary->wait_for_catchup($standby, 'replay', $primary->lsn('flush')); +is($standby->safe_psql('postgres', 'SELECT pg_is_in_recovery()'), + 't', 'node remains a standby'); +is($standby->safe_psql('postgres', 'SELECT count(*) FROM pg_stat_replication'), + '0', 'standby has no downstream replication'); +ok(-f $standby->data_dir . '/pg_wal/00000003.history', + 'backup supplied the current timeline history'); +ok(!-e $standby->data_dir . '/pg_wal/00000002.history', + 'backup did not supply the intermediate timeline history'); + +my $history_command = [ + 'psql', '-XAt', + '--dbname' => $standby->connstr . ' replication=1', + '--command' => 'TIMELINE_HISTORY 2' +]; +my $history_error = qr/could not open file "pg_wal\/00000002\.history"/; +command_fails_like($history_command, $history_error, + 'history fallback is disabled by default'); +is($standby->safe_psql('postgres', $scratch_count), + '0', 'disabled fallback creates no scratch copy'); + +# The current history can still be read locally with fallback disabled. +my $current = slurp_file($standby->data_dir . '/pg_wal/00000003.history'); +my $expected_current = "00000003.history|$current"; +command_like( + [ + 'psql', '-XAt', + '--dbname' => $standby->connstr . ' replication=1', + '--command' => 'TIMELINE_HISTORY 3' + ], + qr/\Q$expected_current\E/, 'current history is served from pg_wal'); + +$standby->append_conf('postgresql.conf', 'wal_sender_restore = on'); +$standby->reload; +$standby->poll_query_until('postgres', + "SELECT current_setting('wal_sender_restore')::boolean") + or BAIL_OUT('history fallback was not enabled'); +rename("$archive/00000002.history", "$archive/00000002.history.saved") + or die "hide history file: $!"; +command_fails_like($history_command, $history_error, + 'a missing archived history file fails the command'); +rename("$archive/00000002.history.saved", "$archive/00000002.history") + or die "restore history file: $!"; +$standby->poll_query_until('postgres', $scratch_count, '0') + or BAIL_OUT('failed history restore left a scratch copy'); + +my $history = slurp_file("$archive/00000002.history"); +my $expected = "00000002.history|$history"; +command_like($history_command, qr/\Q$expected\E/, + 'physical replication connection receives the archived history contents'); + +# Reuse one database replication connection to check repeated cleanup. +command_like( + [ + 'psql', '-XAt', + '--dbname' => $standby->connstr('postgres') . ' replication=database', + '--command' => 'TIMELINE_HISTORY 2', + '--command' => 'TIMELINE_HISTORY 2' + ], + qr/\Q$expected\E.*\Q$expected\E/s, + 'repeated history requests on one database replication connection'); +$standby->poll_query_until('postgres', $scratch_count, '0') + or BAIL_OUT('history requests left scratch copies'); +ok(!-e $standby->data_dir . '/pg_wal/00000002.history', + 'restored history was not installed in pg_wal'); + +# The node continues to receive ordinary streaming replication throughout. +$primary->safe_psql('postgres', 'INSERT INTO history_test VALUES (2)'); +$primary->wait_for_catchup($standby, 'replay', $primary->lsn('insert')); +is($standby->safe_psql('postgres', 'SELECT sum(i) FROM history_test'), + '3', 'ordinary standby continues replaying upstream WAL'); +is($standby->safe_psql('postgres', 'SELECT count(*) FROM pg_stat_replication'), + '0', 'standby still has no downstream replication'); +$standby->stop; +$primary->stop; +done_testing(); -- 2.50.1 (Apple Git-155)
