Hi Shihao,
On Sat, Oct 24, 2026 at 1:04 AM shihao zhong <[email protected]>
wrote:
> In my test it goes away if GetLatestLSN() returns the replay position
> during recovery. What do you think?
I think you're right. Your test fails on master in 3 of 3 runs (the
restored copy has 4|0 instead of 4|1) and passes with the attached
0002, which is your idea: during recovery GetLatestLSN() returns the
replay position instead of the walreceiver's flush position
(walsummarizer.c:879-890 on master), so no summary of the old timeline
goes past the switch point. I'll add you as Reported-by to 0002 in
the next version.
0002 also takes away what Fujii's assertion needs, a summarizer ahead
of replay at promotion. On a cassert build of the series without my
v2 fix from upthread, the standby never summarizes past the recovery
target, and the 004 test from v2 times out in its setup waiting for
such a summary (3 of 3). So with 0002 in, v2's change would only be
defensive and its test would need a different setup; without 0002,
v2 is still needed.
0002 does not help with a summary that is already on disk, so 0001
covers that. In repro-C-upgrade.sh.txt the standby runs master until
the promotion and writes a timeline 1 summary across the switch point;
then the server is restarted on the full series (the two patches
below plus 0001 and 0002). The summary is still there, and 0001
refuses the backup (3 of 3) because it records a truncation; summary
entries carry no LSNs, so the part past the switch can't be dropped.
Restarted on master instead, the restore loses data (3 of 3). I did
not build 0002 alone; its backup path is master's.
Two neighboring cases came up this week: Ayush's summary that starts
exactly at the switch point [1], and Sai Pranav's prior backup that
ends past it [2]. 0001 goes on top of their patches as posted and adds
only the check for this case; with both of theirs and none of ours,
your case still loses (measurements.txt, table 1).
Robert, the open question is 0002. The flush branch dates from the
first commit, and I could not find why flushed rather than replayed;
in the original thread you called what summarization should do during
recovery "very much live" [3]. Is anything relying on standby
summaries ahead of replay? If so, 0001 alone covers this by refusing.
Your test name collides with Ayush's t/014; I ran it as t/016.
[1]
https://postgr.es/m/CAJTYsWV7RThdo4K62G8hC=anpkbr54qwbovyvvw5uvx4+-x...@mail.gmail.com
[2]
https://postgr.es/m/caphduzc9pwm_vjznmhznfsdzklimqc6dkxoozlukzvps19z...@mail.gmail.com
[3]
https://postgr.es/m/ca+tgmoad7igbt46k+jhg6uez56y5svbyq5e7ojckkgd3sst...@mail.gmail.com
Regards,
Manu
From c5021cea747df90f3615353821179d56fdf6e147 Mon Sep 17 00:00:00 2001
From: Manu <[email protected]>
Date: Fri, 23 Oct 2026 22:15:53 -0300
Subject: [PATCH v1 1/2] Refuse truncations from WAL summaries that cross a
timeline switch
A standby summarizes the WAL it has received, ahead of replay. When it
is promoted at a recovery target inside a WAL summary, the summary of
the old timeline that covers the target also covers WAL the promoted
server never replayed. Block references from that WAL only make an
incremental backup copy more than needed, but a relation truncation
recorded there was applied too: it discards the references to blocks
modified before the switch point, and pg_combinebackup silently takes
those blocks from the prior backup.
When a summary of a timeline this server switched away from ends after
the switch point, refuse a limit block other than 0 from it, asking for
a new full backup. Summary entries carry no LSNs, so a truncation
before the switch point in the same file is refused too.
This goes on top of two fixes for neighboring cases: Ayush Tiwari's,
which stops using a summary that starts exactly at the switch point,
and Sai Pranav's, which refuses a prior backup whose WAL extends past
the switch point. t/014 tests Ayush Tiwari's case; t/015 tests this
one.
---
src/backend/backup/basebackup_incremental.c | 43 +++++-
src/bin/pg_combinebackup/meson.build | 2 +
.../t/014_promote_at_summary_start.pl | 106 ++++++++++++++
.../t/015_summary_past_switch.pl | 130 ++++++++++++++++++
4 files changed, 280 insertions(+), 1 deletion(-)
create mode 100644 src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl
create mode 100644 src/bin/pg_combinebackup/t/015_summary_past_switch.pl
diff --git a/src/backend/backup/basebackup_incremental.c b/src/backend/backup/basebackup_incremental.c
index 3995f23ad1..fabad28ed2 100644
--- a/src/backend/backup/basebackup_incremental.c
+++ b/src/backend/backup/basebackup_incremental.c
@@ -266,7 +266,8 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
MemoryContext oldcontext;
List *expectedTLEs;
List *all_wslist,
- *required_wslist = NIL;
+ *required_wslist = NIL,
+ *past_switch_wslist = NIL;
ListCell *lc;
TimeLineHistoryEntry **tlep;
int num_wal_ranges;
@@ -568,6 +569,29 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
*/
required_wslist = list_concat(required_wslist, tli_wslist);
+ /*
+ * A summary that ends after the point where we switched away from
+ * this timeline also covers WAL that this server may never have
+ * replayed: a standby summarizes the WAL it has received, ahead of
+ * replay, and recovery can stop at a recovery target before the end
+ * of it. Block references from that WAL only make us copy more than
+ * needed, but a relation truncation recorded there would discard the
+ * references to blocks modified before the switch. Remember these
+ * summaries, so that the merge below can refuse such a truncation.
+ */
+ if (tle->tli != backup_state->starttli)
+ {
+ ListCell *lc2;
+
+ foreach(lc2, tli_wslist)
+ {
+ WalSummaryFile *ws = lfirst(lc2);
+
+ if (ws->end_lsn > tli_end_lsn)
+ past_switch_wslist = lappend(past_switch_wslist, ws);
+ }
+ }
+
/*
* Timelines earlier than the one in which the prior backup began are
* not relevant.
@@ -587,6 +611,7 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
foreach(lc, required_wslist)
{
WalSummaryFile *ws = lfirst(lc);
+ bool past_switch = list_member_ptr(past_switch_wslist, ws);
WalSummaryIO wsio;
BlockRefTableReader *reader;
RelFileLocator rlocator;
@@ -605,6 +630,22 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib,
while (BlockRefTableReaderNextRelation(reader, &rlocator, &forknum,
&limit_block))
{
+ /*
+ * A limit block of 0 only makes us copy the whole file, which is
+ * safe. Any other limit from a summary that extends past a
+ * timeline switch may describe a truncation that never happened
+ * on this server, and applying it could lose data.
+ */
+ if (past_switch && limit_block != 0 &&
+ limit_block != InvalidBlockNumber)
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("cannot use WAL summary file \"%s\" for an incremental backup",
+ FilePathName(wsio.file)),
+ errdetail("The file extends past the end of timeline %u and records a relation truncation, which may come from WAL that this server did not replay.",
+ ws->tli),
+ errhint("Take a new full backup.")));
+
BlockRefTableSetLimitBlock(ib->brtab, &rlocator,
forknum, limit_block);
diff --git a/src/bin/pg_combinebackup/meson.build b/src/bin/pg_combinebackup/meson.build
index f1ae1b3594..52412381fc 100644
--- a/src/bin/pg_combinebackup/meson.build
+++ b/src/bin/pg_combinebackup/meson.build
@@ -41,6 +41,8 @@ tests += {
't/011_ib_truncation.pl',
't/012_vm_consistency.pl',
't/013_diverged_parent.pl',
+ 't/014_promote_at_summary_start.pl',
+ 't/015_summary_past_switch.pl',
],
}
}
diff --git a/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl b/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl
new file mode 100644
index 0000000000..9e150f43d6
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl
@@ -0,0 +1,106 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+#
+# A standby can summarize WAL that it has received but not yet replayed. If
+# it is promoted at a recovery target that is exactly the start of such a
+# summary, that old-timeline summary covers only WAL that never happened on
+# the promoted server. An incremental backup taken after promotion must not
+# use it: here it holds a VACUUM truncation that would discard the blocks
+# modified on the old timeline before the switch point.
+
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $mode = $ENV{PG_TEST_PG_COMBINEBACKUP_MODE} || '--copy';
+
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+$primary->append_conf('postgresql.conf', <<EOF);
+summarize_wal = on
+autovacuum = off
+EOF
+$primary->start;
+
+# One row per page, as in 013_summary_order.pl. The heap never exceeds six
+# blocks, so it fits in one segment even with the RELSEG_SIZE used by CI.
+my $block_size = $primary->safe_psql('postgres', 'SHOW block_size');
+my $pad_size = $block_size / 2;
+$primary->safe_psql('postgres', <<EOF);
+CREATE TABLE t (id int, pad text STORAGE PLAIN);
+INSERT INTO t SELECT g, repeat('x', $pad_size) FROM generate_series(1, 4) g;
+VACUUM t;
+EOF
+$primary->backup('full');
+my $full_path = $primary->backup_dir . '/full';
+
+# Updating rows 3 and 4 modifies blocks 2 and 3 and extends the heap to six
+# blocks. The standby replays this. The recovery target is the redo pointer
+# of the following checkpoint, so it is also the start of a WAL summary.
+my $size_sql = "SELECT pg_relation_size('t') / $block_size";
+$primary->safe_psql('postgres', "UPDATE t SET pad = pad || 'o' WHERE id > 2");
+is($primary->safe_psql('postgres', $size_sql), '6',
+ 'UPDATE extends the heap to six blocks');
+$primary->safe_psql('postgres', 'CHECKPOINT');
+my $target_lsn = $primary->safe_psql('postgres',
+ 'SELECT redo_lsn FROM pg_control_checkpoint()');
+
+# The standby receives and summarizes this, but never replays it.
+$primary->safe_psql('postgres', 'DELETE FROM t WHERE id > 2');
+$primary->safe_psql('postgres', 'VACUUM t');
+is($primary->safe_psql('postgres', $size_sql), '2',
+ 'VACUUM truncates the heap to two blocks after the recovery target');
+$primary->safe_psql('postgres', 'CHECKPOINT');
+my $end_lsn = $primary->safe_psql('postgres',
+ 'SELECT redo_lsn FROM pg_control_checkpoint()');
+
+my $standby = PostgreSQL::Test::Cluster->new('standby');
+$standby->init_from_backup($primary, 'full', has_streaming => 1);
+$standby->append_conf('postgresql.conf', <<EOF);
+recovery_target_lsn = '$target_lsn'
+recovery_target_inclusive = false
+recovery_target_action = 'pause'
+EOF
+$standby->start;
+$standby->poll_query_until('postgres',
+ "SELECT pg_get_wal_replay_pause_state() = 'paused'")
+ or die "timed out waiting for recovery to pause";
+$standby->poll_query_until('postgres', <<EOF)
+SELECT EXISTS (SELECT FROM pg_available_wal_summaries()
+ WHERE tli = 1 AND start_lsn = '$target_lsn'
+ AND end_lsn >= '$end_lsn')
+EOF
+ or die "timed out waiting for a timeline 1 summary past $target_lsn";
+$primary->stop;
+$standby->promote;
+
+# Don't read the table before the backup: with data checksums, setting hint
+# bits would WAL-log those blocks again on the new timeline and hide the loss.
+# Store both backups under $primary so init_from_backup() can combine them.
+$standby->command_ok(
+ [
+ 'pg_basebackup', '--no-sync',
+ '--pgdata' => $primary->backup_dir . '/incr',
+ '--checkpoint' => 'fast',
+ '--incremental' => "$full_path/backup_manifest",
+ ],
+ 'incremental backup after promotion');
+
+my $contents_sql = "SELECT id, right(pad, 1) FROM t ORDER BY id";
+my $expected = "1|x\n2|x\n3|o\n4|o";
+is($standby->safe_psql('postgres', $contents_sql),
+ $expected, 'promoted standby has the rows updated before the target');
+$standby->stop;
+
+my $restored = PostgreSQL::Test::Cluster->new('restored');
+$restored->init_from_backup(
+ $primary, 'incr',
+ combine_with_prior => ['full'],
+ combine_mode => $mode);
+$restored->start;
+is($restored->safe_psql('postgres', $contents_sql),
+ $expected, 'restored rows match the promoted standby');
+$restored->stop;
+
+done_testing();
diff --git a/src/bin/pg_combinebackup/t/015_summary_past_switch.pl b/src/bin/pg_combinebackup/t/015_summary_past_switch.pl
new file mode 100644
index 0000000000..b1c0bf8c17
--- /dev/null
+++ b/src/bin/pg_combinebackup/t/015_summary_past_switch.pl
@@ -0,0 +1,130 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+#
+# A streaming standby summarizes the WAL it receives, ahead of replay. When
+# it is promoted at a recovery target inside a WAL summary, the old-timeline
+# summary covering the target also covers WAL that the promoted server never
+# replayed. Block references from that WAL are harmless, but a truncation
+# recorded there must not be applied to an incremental backup.
+
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $mode = $ENV{PG_TEST_PG_COMBINEBACKUP_MODE} || '--copy';
+
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+$primary->append_conf('postgresql.conf', <<EOF);
+summarize_wal = on
+autovacuum = off
+EOF
+$primary->start;
+
+# One row per page, so that the heap fits in one segment even with the
+# six-block RELSEG_SIZE used by CI.
+my $block_size = $primary->safe_psql('postgres', 'SHOW block_size');
+my $pad_size = $block_size / 2;
+$primary->safe_psql('postgres', <<EOF);
+CREATE TABLE t (id int, pad text STORAGE PLAIN);
+INSERT INTO t SELECT g, repeat('x', $pad_size) FROM generate_series(1, 4) g;
+VACUUM t;
+EOF
+$primary->backup('full');
+my $full_path = $primary->backup_dir . '/full';
+
+# Each recovery target is inside a WAL summary: after the redo pointer of a
+# checkpoint and some WAL that the standby replays.
+my $insert_lsn = 'SELECT pg_current_wal_insert_lsn()';
+$primary->safe_psql('postgres', 'CHECKPOINT');
+$primary->safe_psql('postgres', "UPDATE t SET pad = pad || 'o' WHERE id > 2");
+my $target1 = $primary->safe_psql('postgres', $insert_lsn);
+
+# Past the first target, only new rows: harmless for that standby.
+$primary->safe_psql('postgres',
+ "INSERT INTO t VALUES (5, repeat('x', $pad_size))");
+$primary->safe_psql('postgres', 'CHECKPOINT');
+$primary->safe_psql('postgres', 'CREATE TABLE marker ()');
+my $target2 = $primary->safe_psql('postgres', $insert_lsn);
+
+# Past the second target, a truncation from 7 blocks to 2.
+$primary->safe_psql('postgres', 'DELETE FROM t WHERE id > 2');
+$primary->safe_psql('postgres', 'VACUUM t');
+is($primary->safe_psql('postgres', "SELECT pg_relation_size('t') / $block_size"),
+ '2', 'VACUUM truncates the heap after the second target');
+$primary->safe_psql('postgres', 'CHECKPOINT');
+my $end_lsn = $primary->safe_psql('postgres',
+ 'SELECT redo_lsn FROM pg_control_checkpoint()');
+
+# Start a standby that pauses at the target and has summarized everything
+# the primary wrote, then promote it.
+sub promoted_standby
+{
+ my ($name, $target) = @_;
+ my $node = PostgreSQL::Test::Cluster->new($name);
+ $node->init_from_backup($primary, 'full', has_streaming => 1);
+ $node->append_conf('postgresql.conf', <<EOF);
+recovery_target_lsn = '$target'
+recovery_target_inclusive = false
+recovery_target_action = 'pause'
+EOF
+ $node->start;
+ $node->poll_query_until('postgres',
+ "SELECT pg_get_wal_replay_pause_state() = 'paused'")
+ or die "timed out waiting for $name to pause";
+ $node->poll_query_until('postgres', <<EOF)
+SELECT EXISTS (SELECT FROM pg_available_wal_summaries()
+ WHERE tli = 1 AND end_lsn >= '$end_lsn')
+EOF
+ or die "timed out waiting for $name to summarize up to $end_lsn";
+ $node->promote;
+ return $node;
+}
+
+my $standby1 = promoted_standby('standby1', $target1);
+my $standby2 = promoted_standby('standby2', $target2);
+$primary->stop;
+
+# The tables are not read before the backups: with data checksums, setting
+# hint bits would WAL-log their blocks again on the new timeline.
+my $contents_sql = "SELECT id, right(pad, 1) FROM t ORDER BY id";
+
+# First standby: the summary past the switch only adds block references, so
+# the incremental backup works and restores the promoted server's rows.
+$standby1->command_ok(
+ [
+ 'pg_basebackup', '--no-sync',
+ '--pgdata' => $primary->backup_dir . '/incr1',
+ '--checkpoint' => 'fast',
+ '--incremental' => "$full_path/backup_manifest",
+ ],
+ 'incremental backup when only new rows follow the target');
+my $expected1 = "1|x\n2|x\n3|o\n4|o";
+is($standby1->safe_psql('postgres', $contents_sql),
+ $expected1, 'first standby has the rows replayed before the target');
+$standby1->stop;
+my $restored = PostgreSQL::Test::Cluster->new('restored');
+$restored->init_from_backup(
+ $primary, 'incr1',
+ combine_with_prior => ['full'],
+ combine_mode => $mode);
+$restored->start;
+is($restored->safe_psql('postgres', $contents_sql),
+ $expected1, 'restored rows match the first standby');
+$restored->stop;
+
+# Second standby: the summary past the switch records the truncation, which
+# never happened on this server, so the incremental backup must refuse it.
+$standby2->command_fails_like(
+ [
+ 'pg_basebackup', '--no-sync',
+ '--pgdata' => $primary->backup_dir . '/incr2',
+ '--checkpoint' => 'fast',
+ '--incremental' => "$full_path/backup_manifest",
+ ],
+ qr/cannot use WAL summary file .* for an incremental backup/,
+ 'incremental backup refuses a truncation past the switch point');
+$standby2->stop;
+
+done_testing();
--
2.56.0
From 80a1a72728589ab40df51acb7e7073d5b03588a5 Mon Sep 17 00:00:00 2001
From: Manu <[email protected]>
Date: Fri, 23 Oct 2026 20:45:19 -0300
Subject: [PATCH v1 2/2] Summarize only replayed WAL on a standby
During recovery, GetLatestLSN() let the WAL summarizer read up to the
walreceiver's flush position whenever that was ahead of replay. That
WAL is not part of the server's history until it is replayed: recovery
can stop at a recovery target before reaching it, and if the server is
then promoted, the summaries already written for the old timeline
cover WAL from a history the server abandoned. Incremental backups
trust those summaries; a relation truncation recorded there discards
the references to blocks modified before the switch point, and the
restored blocks silently come from the prior backup.
Summarize only up to the replay position during recovery. Backups
taken on a standby start at a restartpoint, which is never ahead of
replay, so they need no summaries beyond it.
With this, a standby promoted at a recovery target no longer leaves
old-timeline summaries that cross the switch point, so the tests added
by the previous commit now check that the summarizer stops at replay
and that every incremental backup across the promotion restores the
promoted server's rows.
---
src/backend/postmaster/walsummarizer.c | 35 +++++-------
.../t/014_promote_at_summary_start.pl | 30 ++++++-----
.../t/015_summary_past_switch.pl | 54 ++++++++++++-------
3 files changed, 64 insertions(+), 55 deletions(-)
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 8333a4eaaa..56d24f5323 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -846,10 +846,6 @@ GetLatestLSN(TimeLineID *tli)
}
else
{
- XLogRecPtr flush_lsn;
- TimeLineID flush_tli;
- XLogRecPtr replay_lsn;
- TimeLineID replay_tli;
TimeLineID insert_tli;
/*
@@ -868,26 +864,19 @@ GetLatestLSN(TimeLineID *tli)
}
/*
- * What we really want to know is how much WAL has been flushed to
- * disk, but the only flush position available is the one provided by
- * the walreceiver, which may not be running, because this could be
- * crash recovery or recovery via restore_command. So use either the
- * WAL receiver's flush position or the replay position, whichever is
- * further ahead, on the theory that if the WAL has been replayed then
- * it must also have been flushed to disk.
+ * Otherwise, summarize only WAL that has been replayed. A walreceiver
+ * may have flushed WAL beyond that point, but it is not yet part of
+ * this server's history: recovery may stop at a recovery target before
+ * reaching it, and if the server is then promoted, that WAL belongs to
+ * a timeline this server abandoned. A summary covering it would
+ * describe changes, such as relation truncations, that never happened
+ * here, and incremental backups would trust it.
+ *
+ * Backups taken during recovery start at a restartpoint, which is
+ * never ahead of replay, so they don't need summaries of anything
+ * further.
*/
- flush_lsn = GetWalRcvFlushRecPtr(NULL, &flush_tli);
- replay_lsn = GetXLogReplayRecPtr(&replay_tli);
- if (flush_lsn > replay_lsn)
- {
- *tli = flush_tli;
- return flush_lsn;
- }
- else
- {
- *tli = replay_tli;
- return replay_lsn;
- }
+ return GetXLogReplayRecPtr(tli);
}
}
diff --git a/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl b/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl
index 9e150f43d6..6fc4a6a856 100644
--- a/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl
+++ b/src/bin/pg_combinebackup/t/014_promote_at_summary_start.pl
@@ -1,11 +1,10 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
#
-# A standby can summarize WAL that it has received but not yet replayed. If
-# it is promoted at a recovery target that is exactly the start of such a
-# summary, that old-timeline summary covers only WAL that never happened on
-# the promoted server. An incremental backup taken after promotion must not
-# use it: here it holds a VACUUM truncation that would discard the blocks
-# modified on the old timeline before the switch point.
+# A standby promoted at a recovery target that is exactly the start of a WAL
+# summary, while it has received WAL past the target. A standby only
+# summarizes replayed WAL, so no old-timeline summary covers that WAL, and an
+# incremental backup taken after promotion restores the promoted server's
+# rows, not the effects of the VACUUM truncation it never replayed.
use strict;
use warnings FATAL => 'all';
@@ -46,7 +45,7 @@ $primary->safe_psql('postgres', 'CHECKPOINT');
my $target_lsn = $primary->safe_psql('postgres',
'SELECT redo_lsn FROM pg_control_checkpoint()');
-# The standby receives and summarizes this, but never replays it.
+# The standby receives this, but never replays it.
$primary->safe_psql('postgres', 'DELETE FROM t WHERE id > 2');
$primary->safe_psql('postgres', 'VACUUM t');
is($primary->safe_psql('postgres', $size_sql), '2',
@@ -66,12 +65,17 @@ $standby->start;
$standby->poll_query_until('postgres',
"SELECT pg_get_wal_replay_pause_state() = 'paused'")
or die "timed out waiting for recovery to pause";
-$standby->poll_query_until('postgres', <<EOF)
-SELECT EXISTS (SELECT FROM pg_available_wal_summaries()
- WHERE tli = 1 AND start_lsn = '$target_lsn'
- AND end_lsn >= '$end_lsn')
-EOF
- or die "timed out waiting for a timeline 1 summary past $target_lsn";
+$standby->poll_query_until('postgres',
+ "SELECT pg_last_wal_receive_lsn() >= '$end_lsn'")
+ or die "timed out waiting for the standby to receive up to $end_lsn";
+$standby->poll_query_until('postgres',
+ "SELECT pending_lsn >= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()")
+ or die "timed out waiting for summarization to reach replay";
+is( $standby->safe_psql(
+ 'postgres',
+ "SELECT pending_lsn <= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()"),
+ 't',
+ 'the summarizer does not read past replay');
$primary->stop;
$standby->promote;
diff --git a/src/bin/pg_combinebackup/t/015_summary_past_switch.pl b/src/bin/pg_combinebackup/t/015_summary_past_switch.pl
index b1c0bf8c17..5af1a85e5e 100644
--- a/src/bin/pg_combinebackup/t/015_summary_past_switch.pl
+++ b/src/bin/pg_combinebackup/t/015_summary_past_switch.pl
@@ -1,10 +1,10 @@
# Copyright (c) 2026, PostgreSQL Global Development Group
#
-# A streaming standby summarizes the WAL it receives, ahead of replay. When
-# it is promoted at a recovery target inside a WAL summary, the old-timeline
-# summary covering the target also covers WAL that the promoted server never
-# replayed. Block references from that WAL are harmless, but a truncation
-# recorded there must not be applied to an incremental backup.
+# A streaming standby promoted at a recovery target inside a WAL summary,
+# after receiving WAL past the target, including a truncation. A standby only
+# summarizes replayed WAL, so the old-timeline summaries end at the switch
+# point, and incremental backups after promotion restore the promoted
+# server's rows.
use strict;
use warnings FATAL => 'all';
@@ -57,8 +57,8 @@ $primary->safe_psql('postgres', 'CHECKPOINT');
my $end_lsn = $primary->safe_psql('postgres',
'SELECT redo_lsn FROM pg_control_checkpoint()');
-# Start a standby that pauses at the target and has summarized everything
-# the primary wrote, then promote it.
+# Start a standby that pauses at the target and has received everything the
+# primary wrote, then promote it.
sub promoted_standby
{
my ($name, $target) = @_;
@@ -73,11 +73,17 @@ EOF
$node->poll_query_until('postgres',
"SELECT pg_get_wal_replay_pause_state() = 'paused'")
or die "timed out waiting for $name to pause";
- $node->poll_query_until('postgres', <<EOF)
-SELECT EXISTS (SELECT FROM pg_available_wal_summaries()
- WHERE tli = 1 AND end_lsn >= '$end_lsn')
-EOF
- or die "timed out waiting for $name to summarize up to $end_lsn";
+ $node->poll_query_until('postgres',
+ "SELECT pg_last_wal_receive_lsn() >= '$end_lsn'")
+ or die "timed out waiting for $name to receive up to $end_lsn";
+ $node->poll_query_until('postgres',
+ "SELECT pending_lsn >= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()")
+ or die "timed out waiting for $name to summarize up to replay";
+ is( $node->safe_psql(
+ 'postgres',
+ "SELECT pending_lsn <= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()"),
+ 't',
+ "$name: the summarizer does not read past replay");
$node->promote;
return $node;
}
@@ -90,8 +96,7 @@ $primary->stop;
# hint bits would WAL-log their blocks again on the new timeline.
my $contents_sql = "SELECT id, right(pad, 1) FROM t ORDER BY id";
-# First standby: the summary past the switch only adds block references, so
-# the incremental backup works and restores the promoted server's rows.
+# First standby: only new rows follow its target.
$standby1->command_ok(
[
'pg_basebackup', '--no-sync',
@@ -114,17 +119,28 @@ is($restored->safe_psql('postgres', $contents_sql),
$expected1, 'restored rows match the first standby');
$restored->stop;
-# Second standby: the summary past the switch records the truncation, which
-# never happened on this server, so the incremental backup must refuse it.
-$standby2->command_fails_like(
+# Second standby: the truncation after its target was received but never
+# replayed or summarized, so the incremental backup restores the rows.
+$standby2->command_ok(
[
'pg_basebackup', '--no-sync',
'--pgdata' => $primary->backup_dir . '/incr2',
'--checkpoint' => 'fast',
'--incremental' => "$full_path/backup_manifest",
],
- qr/cannot use WAL summary file .* for an incremental backup/,
- 'incremental backup refuses a truncation past the switch point');
+ 'incremental backup when a truncation follows the target');
+my $expected2 = "1|x\n2|x\n3|o\n4|o\n5|x";
+is($standby2->safe_psql('postgres', $contents_sql),
+ $expected2, 'second standby has the rows replayed before the target');
$standby2->stop;
+my $restored2 = PostgreSQL::Test::Cluster->new('restored2');
+$restored2->init_from_backup(
+ $primary, 'incr2',
+ combine_with_prior => ['full'],
+ combine_mode => $mode);
+$restored2->start;
+is($restored2->safe_psql('postgres', $contents_sql),
+ $expected2, 'restored rows match the second standby');
+$restored2->stop;
done_testing();
--
2.56.0
#!/bin/bash
# Variant for an old summary: everything up to the promotion runs with PGBIN,
# the build without the fix, so the standby writes a timeline 1 summary that
# crosses the switch point. Then the promoted server is restarted with
# PGBIN_NEW (a minor update), which takes the incremental backup, combines
# and compares. PGBIN_NEW=PGBIN is the control (a restart, no update).
#
# Incremental backup after promoting a streaming standby at a recovery target
# that falls inside a WAL summary. The standby's summarizer follows the
# walreceiver, so it has already written an old-timeline summary that runs
# past the switch point, including WAL the promoted server never replayed.
# That summary is used for the old timeline, and what the never-replayed WAL
# did to a relation (here by default a VACUUM truncation) is applied to the
# incremental backup.
#
# usage: PGBIN=/path/to/bin bash repro-C.sh WORKDIR [PORT]
# knobs (environment):
# KIND=lsn recovery_target_lsn = an insert LSN, inclusive = false
[default]
# KIND=lsn-incl the same LSN, recovery_target_inclusive = true
# KIND=xid recovery_target_xid = the xid of the replayed UPDATE
# KIND=time recovery_target_time = a timestamp taken after it
# KIND=name recovery_target_name = a restore point created after it
# ACTION=pause recovery_target_action, then pg_ctl promote [default]
# ACTION=promote recovery_target_action = promote (no wait for summaries)
# JUNK=vacuum never-replayed WAL: DELETE + VACUUM truncates t 8 -> 4
blocks [default]
# JUNK=drop | truncate | vacuumfull | cluster on t
# JUNK=dropdb DROP DATABASE d2 (d2.t2 was updated before the target)
# JUNK=none nothing destructive after the target (control)
# WAIT=summary (default) waits for a timeline 1 summary past the target;
# WAIT=receive only for the walreceiver, then for the summarizer to reach
replay.
# exit: 0 restored data matches the source, 1 it does not, 2 setup problem,
# 3 the incremental backup was refused
set -u
: "${PGBIN:?set PGBIN to the bin directory of the build under test}"
: "${PGBIN_NEW:?set PGBIN_NEW to the bin directory used after the update}"
W=${1:?usage: PGBIN=... bash repro-C.sh WORKDIR [PORT]}
P=${2:-5440}; P1=$((P + 1)); P2=$((P + 2))
KIND=${KIND:-lsn}; ACTION=${ACTION:-pause}; JUNK=${JUNK:-vacuum}
die() { echo "SETUP: $*"; exit 2; }
q() { "$PGBIN/psql" -X -h "$W" -p "$1" -d "${3:-postgres}" -Atq -c "$2"; }
poll() { # poll PORT SQL: wait up to 60 s for SQL to return t
for _ in $(seq 600); do [ "$(q "$1" "$2" 2>/dev/null)" = t ] && return 0;
sleep 0.1; done
return 1
}
stopall() { for d in src standby restored; do
[ -f "$W/$d/postmaster.pid" ] && "$PGBIN/pg_ctl" -D "$W/$d" -m immediate -w
stop >/dev/null 2>&1
done; }
trap stopall EXIT
rm -rf "$W"; mkdir -p "$W" || die "mkdir $W"
"$PGBIN/initdb" -D "$W/src" -A trust --no-locale >/dev/null || die initdb
cat >> "$W/src/postgresql.conf" <<EOC
port = $P
unix_socket_directories = '$W'
listen_addresses = ''
summarize_wal = on
autovacuum = off
EOC
"$PGBIN/pg_ctl" -D "$W/src" -l "$W/src.log" -w start >/dev/null || die "start
primary"
# One row per page: t has 8 rows in 8 blocks; d2.t2 the same.
q $P "CREATE TABLE t (id int, pad text STORAGE PLAIN)"
q $P "INSERT INTO t SELECT g, repeat('x', 4096) FROM generate_series(1, 8) g"
q $P "CREATE INDEX t_id ON t (id)"
q $P "VACUUM t"
q $P "CREATE DATABASE d2"
q $P "CREATE TABLE t2 (id int, pad text STORAGE PLAIN)" d2
q $P "INSERT INTO t2 SELECT g, repeat('y', 4096) FROM generate_series(1, 8) g"
d2
q $P "VACUUM t2" d2
"$PGBIN/pg_basebackup" -h "$W" -p $P -D "$W/full" -c fast || die "full backup"
cp -a "$W/full" "$W/standby"
touch "$W/standby/standby.signal"
# A WAL summary starts at this checkpoint's redo LSN; the target falls after it.
q $P "CHECKPOINT"
SUMSTART=$(q $P "SELECT redo_lsn FROM pg_control_checkpoint()")
# Replayed by the standby: new versions of rows 5..8 (t blocks 4..11).
UPDXID=$(q $P "BEGIN; UPDATE t SET pad = pad || 'o' WHERE id > 4;
SELECT pg_current_xact_id(); COMMIT;")
q $P "UPDATE t2 SET pad = pad || 'o' WHERE id > 4" d2
sleep 1
TS=$(q $P "SELECT clock_timestamp()")
sleep 1
q $P "SELECT pg_create_restore_point('c_target')" >/dev/null
q $P "CREATE TABLE marker ()"
LSN=$(q $P "SELECT pg_current_wal_insert_lsn()")
case $KIND in
lsn) TGT="recovery_target_lsn = '$LSN'
recovery_target_inclusive = false" ;;
lsn-incl) TGT="recovery_target_lsn = '$LSN'
recovery_target_inclusive = true" ;;
xid) TGT="recovery_target_xid = '$UPDXID'" ;;
time) TGT="recovery_target_time = '$TS'" ;;
name) TGT="recovery_target_name = 'c_target'" ;;
*) die "KIND=$KIND" ;;
esac
# Received by the standby, never replayed.
case $JUNK in
vacuum) q $P "DELETE FROM t WHERE id > 4"; q $P "VACUUM t" ;;
drop) q $P "DROP TABLE t" ;;
truncate) q $P "TRUNCATE t" ;;
vacuumfull) q $P "VACUUM FULL t" ;;
cluster) q $P "CLUSTER t USING t_id" ;;
dropdb) q $P "DROP DATABASE d2" ;;
none) q $P "CREATE TABLE marker2 ()" ;;
*) die "JUNK=$JUNK" ;;
esac
q $P "CHECKPOINT"
X=$(q $P "SELECT redo_lsn FROM pg_control_checkpoint()")
echo "KIND=$KIND ACTION=$ACTION JUNK=$JUNK; summary start $SUMSTART, last
primary redo $X"
cat >> "$W/standby/postgresql.conf" <<EOC
port = $P1
primary_conninfo = 'host=$W port=$P'
$TGT
recovery_target_action = '$ACTION'
EOC
"$PGBIN/pg_ctl" -D "$W/standby" -l "$W/standby.log" -w start >/dev/null || die
"start standby"
if [ "$ACTION" = pause ]; then
poll $P1 "SELECT pg_get_wal_replay_pause_state() = 'paused'" || die "standby
did not pause"
# The summarizer follows the walreceiver, not replay.
if [ "${WAIT:-summary}" = summary ]; then
poll $P1 "SELECT EXISTS (SELECT FROM pg_available_wal_summaries()
WHERE tli = 1 AND end_lsn >= '$X')" || die "no timeline 1 summary
up to $X"
else
# WAIT=receive: the standby has received everything; summarization has
# caught up with replay (a summarizer that follows replay stops there).
poll $P1 "SELECT pg_last_wal_receive_lsn() >= '$X'" || die "standby did not
receive up to $X"
poll $P1 "SELECT pending_lsn >= pg_last_wal_replay_lsn() FROM
pg_get_wal_summarizer_state()" \
|| die "summarizer did not reach replay"
sleep 2
echo "summarizer pending $(q $P1 "SELECT pending_lsn FROM
pg_get_wal_summarizer_state()"), replay $(q $P1 "SELECT
pg_last_wal_replay_lsn()")"
fi
"$PGBIN/pg_ctl" -D "$W/src" -w stop >/dev/null
"$PGBIN/pg_ctl" -D "$W/standby" -w promote >/dev/null || die promote
else
poll $P1 "SELECT NOT pg_is_in_recovery()" || die "standby did not promote"
"$PGBIN/pg_ctl" -D "$W/src" -w stop >/dev/null
fi
SW=$(awk '$1 == 1 {print $2}' "$W/standby/pg_wal/00000002.history")
echo "switch point $SW; timeline 1 summaries on the standby:"
q $P1 "SELECT ' ' || start_lsn || ' .. ' || end_lsn ||
CASE WHEN start_lsn < '$SW' AND end_lsn > '$SW' THEN ' <- crosses the
switch point'
WHEN start_lsn = '$SW' THEN ' <- starts at the switch point' ELSE
'' END
FROM pg_available_wal_summaries() WHERE tli = 1 ORDER BY start_lsn"
q $P1 "CHECKPOINT"
# Minor update: restart the promoted server with the new binaries.
"$PGBIN/pg_ctl" -D "$W/standby" -w stop >/dev/null || die "stop before update"
PGBIN=$PGBIN_NEW
"$PGBIN/pg_ctl" -D "$W/standby" -l "$W/standby-new.log" -w start >/dev/null ||
die "start after update"
echo "restarted with $PGBIN ($("$PGBIN/postgres" -V))"
q $P1 "SELECT ' after update: ' || start_lsn || ' .. ' || end_lsn FROM
pg_available_wal_summaries() WHERE tli = 1 AND start_lsn < '$SW' AND end_lsn >
'$SW'"
# Don't read the tables before the backup: with data checksums, setting hint
# bits WAL-logs those blocks again on the new timeline and hides the loss.
if ! "$PGBIN/pg_basebackup" -h "$W" -p $P1 -D "$W/incr" -c fast \
--incremental "$W/full/backup_manifest"; then echo "RESULT: REFUSED"; exit 3;
fi
FP="SELECT count(*) || '|' || coalesce(max(id), 0) || '|' ||
md5(coalesce(string_agg(id
|| ':' || md5(pad), ',' ORDER BY id), ''))"
fp() { echo "t $(q $1 "$FP FROM t") t2 $(q $1 "$FP FROM t2" d2)"; }
REL=$(q $P1 "SELECT pg_relation_filepath('t')")
SRC=$(fp $P1)
"$PGBIN/pg_ctl" -D "$W/standby" -w stop >/dev/null
echo "timeline 1 summaries with an entry for t ($REL):"
for f in $(ls "$W/standby/pg_wal/summaries" | grep '^00000001'); do
s=$("$PGBIN/pg_walsummary" "$W/standby/pg_wal/summaries/$f" \
| grep "REL ${REL##*/}, FORK main" | sed 's/.*FORK main: //' | tr '\n' '
')
echo " $f: ${s:--}"
done
"$PGBIN/pg_verifybackup" -q "$W/full" && "$PGBIN/pg_verifybackup" -q "$W/incr" \
&& echo "pg_verifybackup: both backups OK"
"$PGBIN/pg_combinebackup" -o "$W/restored" "$W/full" "$W/incr" || die
pg_combinebackup
sed -i "s/^port = $P1\$/port = $P2/; /^recovery_target/d; /^primary_conninfo/d"
\
"$W/restored/postgresql.conf"
"$PGBIN/pg_ctl" -D "$W/restored" -l "$W/restored.log" -w start >/dev/null ||
die "start restored"
DST=$(fp $P2)
"$PGBIN/pg_ctl" -D "$W/restored" -w stop >/dev/null
echo "source: $SRC (count|max(id)|md5 of rows)"
echo "restored: $DST"
if [ "$SRC" = "$DST" ]; then echo "RESULT: OK"; exit 0; fi
echo "RESULT: DATA LOSS"; exit 1
Incremental backup after a timeline switch: measurements
=========================================================
Production builds (no assertions), Linux x86_64, data directories on tmpfs.
Base for every row below: master 126976a8ff plus Fujii's v1 for #19774
("Fix incremental backup data loss due to WAL summary merge"), which
merges summaries in LSN order. On plain master, with summaries merged in
readdir order, case A happened to restore correctly (0 of 3) while case C
lost (3 of 3).
Cases
A a timeline 1 summary that starts exactly at the switch point
(Ayush's report); repro-A.sh.txt
C a timeline 1 summary that starts before the switch point and ends
after it, with a VACUUM truncation after the recovery target
(this report); repro-C.sh.txt
R a prior backup whose WAL range ends after the switch point
(Sai Pranav's report); his t/013_diverged_parent.pl, tests 3-4
Outcome per run: "ok" = restored rows equal the promoted server's rows,
"LOSS" = they differ, "refused" = the incremental backup errors out.
R: "accepted" = his tests 3-4 fail (the backup succeeds), "refused" =
they pass.
1. Each case against each patch (3 runs per cell)
patches on top of the base case A case C case R
----------------------------- ------------ ------------ --------
none LOSS 3/3 LOSS 3/3 accepted
Ayush's v1 ok 3/3 LOSS 3/3 accepted
Sai Pranav's v1 LOSS 3/3 LOSS 3/3 refused
Ayush's v1 + Sai Pranav's v1 ok 3/3 LOSS 3/3 refused
+ 0001 ok 3/3 refused 3/3 refused
+ 0001 + 0002 ok 3/3 (*) ok 3/3 (*) refused
0001 and 0002 are applied on top of Ayush's and Sai Pranav's v1 patches
as posted.
(*) With 0002 the precondition of both scripts never appears: no
timeline 1 summary goes past the recovery target before promotion
(default WAIT=summary gives up waiting for one, 1 of 1). These
cells use WAIT=receive, which waits until the standby has received
all the WAL and summarization has reached replay. On the base
without patches WAIT=receive loses too: A 3/3, C 3/3.
Controls, case C with nothing destructive after the target
(JUNK=none): restores with 0001 (1/1) and with 0001 + 0002 (1/1).
t/014 and t/015 pass on 0001 (11 tests) and on 0001 + 0002 (15 tests).
2. Case C without patches (repro-C.sh.txt, default WAIT=summary)
Branches, plain heads of today, 5 runs each:
REL_19_STABLE fc189eb7508, REL_18_STABLE 9412c35035a,
REL_17_STABLE e05b7e2a45c: LOSS 5/5 each.
On the base (master+F):
recovery target kinds (KIND=lsn-incl, xid, time, name): LOSS 3/3 each
(the default KIND=lsn is the 3/3 of table 1)
recovery_target_action = promote (ACTION=promote): ok 5/5; replay
reaches the target before a summary past it is written, so it needs
a pause at the target, which is the default action
other operations after the target (JUNK=drop, truncate, vacuumfull,
cluster, dropdb, none): ok 2/2 each; they leave a limit block of 0,
which copies the whole file, or only extra block references
3. What 0002 costs: incremental backups taken ON A STANDBY (lag.sh.txt)
Full backup taken on the standby; then two batches of changes, each
followed by a checkpoint on the primary; the incremental backup is
taken on the standby when it has replayed the first checkpoint and:
none has also replayed the second batch
delay has received it, but recovery_min_apply_delay = 20s holds it
pause has received it, with replay paused before it
The restored table must be one of the three states of the primary's
history. Base = master+F as above, 3 runs per cell, elapsed time of
pg_basebackup --incremental (coarse: shared build lane, not a clean
benchmark lane).
mode base: result elapsed (s) + all four: result elapsed (s)
------ ------------ ------------------- ------------------ -----------
none ok 3/3 0.115 0.135 0.138 ok 3/3 0.136 0.137 0.051
delay ok 3/3 0.084 0.076 0.078 ok 3/3 0.060 0.058 0.064
pause ok 3/3 0.092 0.081 0.078 ok 3/3 0.059 0.070 0.064
("all four" = Ayush's v1 + Sai Pranav's v1 + 0001 + 0002.)
The backup starts at a restartpoint, which is never ahead of replay,
so it never needed summaries beyond replay. In these runs the
summarizer's position just before the backup was similar in both
builds (its polling interval dominates); with 0002 it cannot pass
replay by construction, which t/014 and t/015 check.
4. Tests (assertions + injection points, master 126976a8ff, with Ayush's
and Sai Pranav's v1 patches underneath)
0001 (with the summarizer fix of the "Assertion failure in WAL
summarizer" thread underneath, which t/014 and t/015 need on
assertion-enabled builds): make check, pg_walsummary and
pg_combinebackup (t/008, t/013, t/014, t/015) pass. With 0001's
basebackup_incremental.c change reverted, t/015 fails; t/013 and t/014
keep passing, as they test the patches underneath.
0001 + 0002 (without that summarizer fix): make check, pg_walsummary,
pg_combinebackup and src/test/recovery pass, with no assertion failure
in the pg_combinebackup logs. With 0002 reverted, t/014 and t/015 fail.
5. Shihao's test, and a summary written before a minor update (25 Oct)
Production builds of master 126976a8ff ("master") and of master +
Ayush's v1 + Sai Pranav's v1 + 0001 + 0002 ("the series"), without
Fujii's #19774 fix underneath (it does not change case C, see 1).
Shihao's t/014_truncation_after_target.pl as posted, 3 runs each:
master: fails 3/3, test 2 'restored backup has the update'
got "4|0", expected "4|1"
the series: passes 3/3 (copied as t/016, t/014 is taken)
Old summary (repro-C-upgrade.sh.txt, default knobs, 3 runs each):
everything up to the promotion runs on master, so the standby writes
the timeline 1 summary 0/03000028 .. 0/0303E500 across the switch
point 0/03034520 ("limit 4" for t). The promoted server is then
restarted and takes the incremental backup.
restarted on master (control): DATA LOSS 3/3
restarted on the series: refused 3/3, with
ERROR: cannot use WAL summary file "pg_wal/summaries/
000000010000000003000028000000000303E500.summary" for an
incremental backup
The same LSNs in all 6 runs. After the restart on the series, with
0002 running, that summary is still listed: 0002 changes where the
summarizer stops, not the summaries already on disk. 0002 alone was
not built; its only non-test change is in walsummarizer.c, so its
backup path is master's and, from reading the code, it would behave
like the control.
