Hi Sai Pranav, On Mon, Nov 2, 2026 at 11:56 PM Raja Sai pranav <[email protected]> wrote: > I tried the series (v2-0001 from upthread plus your v1-0001 and v1-0002) on > current master.
Thanks for testing it on aarch64. v3 is attached, rebased on f549ecd2660 and 316861101e8; v1's t/014 is dropped, as f549ecd2660 covers that case. > - pg_walsummary's 004_promote_past_target times out in its setup with 0002 > applied, as you said: the standby never summarizes past the target. 0001 keeps the v2 test, which trips the assertion on master. 0003 (formerly 0002) makes it promote while the summarizer waits at replay, since the v2 check is then only defensive. On today's master without cassert, Shihao's test (now t/015) still restores 4|0 instead of 4|1, 3 of 3; with the series it passes 3 of 3. 0003 now lists Shihao as co-author, since the idea and t/015 are his. Fujii, v3 is on top of what you committed, in case it helps. Robert, for 0003: does anything rely on summaries ahead of replay? Regards, Manu
From c2e0532fc1d453e150cb5af3226b3f33aae0484a Mon Sep 17 00:00:00 2001 From: Manu <[email protected]> Date: Thu, 22 Oct 2026 11:15:44 -0300 Subject: [PATCH v3 1/3] Fix assertion failure in WAL summarizer after promotion A standby's WAL summarizer summarizes up to what the walreceiver has flushed, which can be well past the point where replay stops at a recovery target. If the standby is then promoted, the old timeline ends at a switch point that the summarizer has already passed. When the summarizer notices the new timeline while waiting for WAL inside SummarizeWAL(), summarizer_read_local_xlog_page() moves read_upto backwards to the switch point, as its comments anticipate, and the next read hits end of WAL. SummarizeWAL() then reported read_upto as the end of the summary, an LSN before the one it started from, and WalSummarizerMain() failed Assert(end_of_summary_lsn >= current_lsn) Builds without assertions were not affected in practice: the main loop treated the bogus end LSN as its new position, noticed the timeline switch on its next iteration and resumed from the switch point on the new timeline. Fix by having SummarizeWAL() report no progress in that case. The main loop then backs up to the switch point through the code that already exists for that purpose. Add a test reproducing the scenario reported by Fujii Masao. Reported-by: Fujii Masao <[email protected]> Reviewed-by: Daria Lepikhova <[email protected]> Tested-by: Daria Lepikhova <[email protected]> Tested-by: Sai Pranav <[email protected]> Discussion: https://postgr.es/m/cahgqgwfbpzr2wkrku9j8w6e+-rzd6ovazs8n_kxvzfccfnj...@mail.gmail.com Backpatch-through: 17 --- src/backend/postmaster/walsummarizer.c | 16 +++- src/bin/pg_walsummary/meson.build | 1 + .../t/004_promote_past_target.pl | 77 +++++++++++++++++++ 3 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 src/bin/pg_walsummary/t/004_promote_past_target.pl diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 8333a4eaaa..88b6fee82b 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -1287,6 +1287,18 @@ SummarizeWAL(TimeLineID tli, XLogRecPtr start_lsn, bool exact, break; } + /* + * If the timeline we were reading became historic while we were waiting + * for more WAL, summarizer_read_local_xlog_page() may have moved read_upto + * backwards to a switch point that precedes the point where this summary + * started (for example, when a standby that has summarized WAL beyond its + * recovery target is promoted). In that case we have summarized nothing, + * so report that we made no progress. The caller will notice the timeline + * switch on its next iteration and back up to the switch point. + */ + if (summary_end_lsn < summary_start_lsn) + summary_end_lsn = summary_start_lsn; + /* Destroy xlogreader. */ pfree(xlogreader->private_data); XLogReaderFree(xlogreader); @@ -1729,7 +1741,9 @@ summarizer_read_local_xlog_page(XLogReaderState *state, * backwards, because we might have been promoted before * reaching the end of the previous timeline. In that * case, the next loop iteration will likely conclude that - * we've reached end of WAL. + * we've reached end of WAL. If the switch point precedes + * the start of the current summary, SummarizeWAL() then + * reports no progress. */ private_data->read_upto = switchpoint; diff --git a/src/bin/pg_walsummary/meson.build b/src/bin/pg_walsummary/meson.build index 839b644ef2..46e02070cb 100644 --- a/src/bin/pg_walsummary/meson.build +++ b/src/bin/pg_walsummary/meson.build @@ -26,6 +26,7 @@ tests += { 't/001_basic.pl', 't/002_blocks.pl', 't/003_tli_switch.pl', + 't/004_promote_past_target.pl', ], } } diff --git a/src/bin/pg_walsummary/t/004_promote_past_target.pl b/src/bin/pg_walsummary/t/004_promote_past_target.pl new file mode 100644 index 0000000000..7672183c73 --- /dev/null +++ b/src/bin/pg_walsummary/t/004_promote_past_target.pl @@ -0,0 +1,77 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group +# +# A standby that stops at a recovery target can have summarized WAL beyond +# that target, because the WAL summarizer reads whatever the walreceiver has +# flushed. When such a standby is promoted, the old timeline ends at a point +# the summarizer has already passed. The summarizer must back up to the switch +# point and continue on the new timeline, rather than report a summary that +# ends before it started (which used to trip an assertion). + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 1); +$primary->append_conf('postgresql.conf', <<EOM); +autovacuum = off +summarize_wal = on +EOM +$primary->start; + +$primary->safe_psql('postgres', 'CREATE TABLE t (i int)'); +$primary->backup('backup'); + +# The recovery target is the redo pointer of a checkpoint, so that the +# standby's summarizer completes a summary that starts exactly there before +# promotion; a second checkpoint lets it summarize past the target. +$primary->safe_psql('postgres', 'INSERT INTO t VALUES (1); CHECKPOINT;'); +my $target_lsn = $primary->safe_psql('postgres', + 'SELECT redo_lsn FROM pg_control_checkpoint()'); +$primary->safe_psql('postgres', 'INSERT INTO t VALUES (2); CHECKPOINT;'); + +my $standby = PostgreSQL::Test::Cluster->new('standby'); +$standby->init_from_backup($primary, 'backup', has_streaming => 1); +$standby->append_conf('postgresql.conf', <<EOM); +recovery_target_lsn = '$target_lsn' +recovery_target_inclusive = false +recovery_target_action = 'pause' +log_min_messages = debug1 +EOM +$standby->start; + +# Wait until replay is paused at the target, the summarizer has summarized +# past it on timeline 1, and the summarizer is waiting for more WAL. +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for recovery to pause"; +$standby->poll_query_until('postgres', <<EOM) +SELECT EXISTS (SELECT FROM pg_available_wal_summaries() + WHERE tli = 1 AND start_lsn = '$target_lsn') +EOM + or die "timed out waiting for summary starting at $target_lsn"; +$standby->poll_query_until('postgres', <<EOM) +SELECT EXISTS (SELECT FROM pg_stat_activity + WHERE backend_type = 'walsummarizer' + AND wait_event = 'WalSummarizerWal') +EOM + or die "timed out waiting for summarizer to wait for WAL"; + +my $log_offset = -s $standby->logfile; +$standby->promote; +$standby->safe_psql('postgres', 'CHECKPOINT'); + +# Summarization must resume on timeline 2 exactly at the switch point. +my $result = $standby->poll_query_until('postgres', <<EOM); +SELECT EXISTS (SELECT FROM pg_available_wal_summaries() + WHERE tli = 2 AND start_lsn = '$target_lsn') +EOM +ok($result, "summarization resumed on timeline 2 at $target_lsn"); + +ok( !$standby->log_contains( + qr/WAL summarizer process .* was terminated by signal/, $log_offset), + 'no WAL summarizer crash after promotion'); + +done_testing(); -- 2.56.0
From 35c7dacbbfb302086fcf298e17172f5203a8e996 Mon Sep 17 00:00:00 2001 From: Manu <[email protected]> Date: Fri, 23 Oct 2026 22:15:53 -0300 Subject: [PATCH v3 2/3] Refuse truncations from WAL summaries that cross a timeline switch A standby summarizes the WAL it has received, ahead of replay. When it is promoted at a recovery target inside a WAL summary, the summary of the old timeline that covers the target also covers WAL the promoted server never replayed. Block references from that WAL only make an incremental backup copy more than needed, but a relation truncation recorded there was applied too: it discards the references to blocks modified before the switch point, and pg_combinebackup silently takes those blocks from the prior backup. When a summary of a timeline this server switched away from ends after the switch point, refuse a limit block other than 0 from it, asking for a new full backup. Summary entries carry no LSNs, so a truncation before the switch point in the same file is refused too. Commit f549ecd2660 stopped using an old-timeline summary that starts exactly at the switch point, and commit 316861101e8 refuses a prior backup whose WAL range extends past it. This covers a summary that starts before the switch point and ends after it. Tested-by: Sai Pranav <[email protected]> Discussion: https://postgr.es/m/cahgqgwfbpzr2wkrku9j8w6e+-rzd6ovazs8n_kxvzfccfnj...@mail.gmail.com --- src/backend/backup/basebackup_incremental.c | 43 +++++- src/bin/pg_combinebackup/meson.build | 1 + .../t/014_summary_past_switch.pl | 130 ++++++++++++++++++ 3 files changed, 173 insertions(+), 1 deletion(-) create mode 100644 src/bin/pg_combinebackup/t/014_summary_past_switch.pl diff --git a/src/backend/backup/basebackup_incremental.c b/src/backend/backup/basebackup_incremental.c index 233da98043..50b5fcf5ff 100644 --- a/src/backend/backup/basebackup_incremental.c +++ b/src/backend/backup/basebackup_incremental.c @@ -267,7 +267,8 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib, MemoryContext oldcontext; List *expectedTLEs; List *all_wslist, - *required_wslist = NIL; + *required_wslist = NIL, + *past_switch_wslist = NIL; ListCell *lc; TimeLineHistoryEntry **tlep; int num_wal_ranges; @@ -566,6 +567,29 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib, */ required_wslist = list_concat(required_wslist, tli_wslist); + /* + * A summary that ends after the point where we switched away from + * this timeline also covers WAL that this server may never have + * replayed: a standby summarizes the WAL it has received, ahead of + * replay, and recovery can stop at a recovery target before the end + * of it. Block references from that WAL only make us copy more than + * needed, but a relation truncation recorded there would discard the + * references to blocks modified before the switch. Remember these + * summaries, so that the merge below can refuse such a truncation. + */ + if (tle->tli != backup_state->starttli) + { + ListCell *lc2; + + foreach(lc2, tli_wslist) + { + WalSummaryFile *ws = lfirst(lc2); + + if (ws->end_lsn > tli_end_lsn) + past_switch_wslist = lappend(past_switch_wslist, ws); + } + } + /* * Timelines earlier than the one in which the prior backup began are * not relevant. @@ -591,6 +615,7 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib, foreach(lc, required_wslist) { WalSummaryFile *ws = lfirst(lc); + bool past_switch = list_member_ptr(past_switch_wslist, ws); WalSummaryIO wsio; BlockRefTableReader *reader; RelFileLocator rlocator; @@ -609,6 +634,22 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib, while (BlockRefTableReaderNextRelation(reader, &rlocator, &forknum, &limit_block)) { + /* + * A limit block of 0 only makes us copy the whole file, which is + * safe. Any other limit from a summary that extends past a + * timeline switch may describe a truncation that never happened + * on this server, and applying it could lose data. + */ + if (past_switch && limit_block != 0 && + limit_block != InvalidBlockNumber) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot use WAL summary file \"%s\" for an incremental backup", + FilePathName(wsio.file)), + errdetail("The file extends past the end of timeline %u and records a relation truncation, which may come from WAL that this server did not replay.", + ws->tli), + errhint("Take a new full backup."))); + BlockRefTableSetLimitBlock(ib->brtab, &rlocator, forknum, limit_block); diff --git a/src/bin/pg_combinebackup/meson.build b/src/bin/pg_combinebackup/meson.build index 10702f36a5..1d5f35af06 100644 --- a/src/bin/pg_combinebackup/meson.build +++ b/src/bin/pg_combinebackup/meson.build @@ -41,6 +41,7 @@ tests += { 't/011_ib_truncation.pl', 't/012_vm_consistency.pl', 't/013_summary_order.pl', + 't/014_summary_past_switch.pl', ], } } diff --git a/src/bin/pg_combinebackup/t/014_summary_past_switch.pl b/src/bin/pg_combinebackup/t/014_summary_past_switch.pl new file mode 100644 index 0000000000..b1c0bf8c17 --- /dev/null +++ b/src/bin/pg_combinebackup/t/014_summary_past_switch.pl @@ -0,0 +1,130 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group +# +# A streaming standby summarizes the WAL it receives, ahead of replay. When +# it is promoted at a recovery target inside a WAL summary, the old-timeline +# summary covering the target also covers WAL that the promoted server never +# replayed. Block references from that WAL are harmless, but a truncation +# recorded there must not be applied to an incremental backup. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $mode = $ENV{PG_TEST_PG_COMBINEBACKUP_MODE} || '--copy'; + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 1); +$primary->append_conf('postgresql.conf', <<EOF); +summarize_wal = on +autovacuum = off +EOF +$primary->start; + +# One row per page, so that the heap fits in one segment even with the +# six-block RELSEG_SIZE used by CI. +my $block_size = $primary->safe_psql('postgres', 'SHOW block_size'); +my $pad_size = $block_size / 2; +$primary->safe_psql('postgres', <<EOF); +CREATE TABLE t (id int, pad text STORAGE PLAIN); +INSERT INTO t SELECT g, repeat('x', $pad_size) FROM generate_series(1, 4) g; +VACUUM t; +EOF +$primary->backup('full'); +my $full_path = $primary->backup_dir . '/full'; + +# Each recovery target is inside a WAL summary: after the redo pointer of a +# checkpoint and some WAL that the standby replays. +my $insert_lsn = 'SELECT pg_current_wal_insert_lsn()'; +$primary->safe_psql('postgres', 'CHECKPOINT'); +$primary->safe_psql('postgres', "UPDATE t SET pad = pad || 'o' WHERE id > 2"); +my $target1 = $primary->safe_psql('postgres', $insert_lsn); + +# Past the first target, only new rows: harmless for that standby. +$primary->safe_psql('postgres', + "INSERT INTO t VALUES (5, repeat('x', $pad_size))"); +$primary->safe_psql('postgres', 'CHECKPOINT'); +$primary->safe_psql('postgres', 'CREATE TABLE marker ()'); +my $target2 = $primary->safe_psql('postgres', $insert_lsn); + +# Past the second target, a truncation from 7 blocks to 2. +$primary->safe_psql('postgres', 'DELETE FROM t WHERE id > 2'); +$primary->safe_psql('postgres', 'VACUUM t'); +is($primary->safe_psql('postgres', "SELECT pg_relation_size('t') / $block_size"), + '2', 'VACUUM truncates the heap after the second target'); +$primary->safe_psql('postgres', 'CHECKPOINT'); +my $end_lsn = $primary->safe_psql('postgres', + 'SELECT redo_lsn FROM pg_control_checkpoint()'); + +# Start a standby that pauses at the target and has summarized everything +# the primary wrote, then promote it. +sub promoted_standby +{ + my ($name, $target) = @_; + my $node = PostgreSQL::Test::Cluster->new($name); + $node->init_from_backup($primary, 'full', has_streaming => 1); + $node->append_conf('postgresql.conf', <<EOF); +recovery_target_lsn = '$target' +recovery_target_inclusive = false +recovery_target_action = 'pause' +EOF + $node->start; + $node->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for $name to pause"; + $node->poll_query_until('postgres', <<EOF) +SELECT EXISTS (SELECT FROM pg_available_wal_summaries() + WHERE tli = 1 AND end_lsn >= '$end_lsn') +EOF + or die "timed out waiting for $name to summarize up to $end_lsn"; + $node->promote; + return $node; +} + +my $standby1 = promoted_standby('standby1', $target1); +my $standby2 = promoted_standby('standby2', $target2); +$primary->stop; + +# The tables are not read before the backups: with data checksums, setting +# hint bits would WAL-log their blocks again on the new timeline. +my $contents_sql = "SELECT id, right(pad, 1) FROM t ORDER BY id"; + +# First standby: the summary past the switch only adds block references, so +# the incremental backup works and restores the promoted server's rows. +$standby1->command_ok( + [ + 'pg_basebackup', '--no-sync', + '--pgdata' => $primary->backup_dir . '/incr1', + '--checkpoint' => 'fast', + '--incremental' => "$full_path/backup_manifest", + ], + 'incremental backup when only new rows follow the target'); +my $expected1 = "1|x\n2|x\n3|o\n4|o"; +is($standby1->safe_psql('postgres', $contents_sql), + $expected1, 'first standby has the rows replayed before the target'); +$standby1->stop; +my $restored = PostgreSQL::Test::Cluster->new('restored'); +$restored->init_from_backup( + $primary, 'incr1', + combine_with_prior => ['full'], + combine_mode => $mode); +$restored->start; +is($restored->safe_psql('postgres', $contents_sql), + $expected1, 'restored rows match the first standby'); +$restored->stop; + +# Second standby: the summary past the switch records the truncation, which +# never happened on this server, so the incremental backup must refuse it. +$standby2->command_fails_like( + [ + 'pg_basebackup', '--no-sync', + '--pgdata' => $primary->backup_dir . '/incr2', + '--checkpoint' => 'fast', + '--incremental' => "$full_path/backup_manifest", + ], + qr/cannot use WAL summary file .* for an incremental backup/, + 'incremental backup refuses a truncation past the switch point'); +$standby2->stop; + +done_testing(); -- 2.56.0
From 2ef278fae2d8c82add5d3514b49f2143b18d44ca Mon Sep 17 00:00:00 2001 From: Manu <[email protected]> Date: Tue, 3 Nov 2026 02:31:11 -0300 Subject: [PATCH v3 3/3] Summarize only replayed WAL on a standby During recovery, GetLatestLSN() let the WAL summarizer read up to the walreceiver's flush position whenever that was ahead of replay. That WAL is not part of the server's history until it is replayed: recovery can stop at a recovery target before reaching it, and if the server is then promoted, the summaries already written for the old timeline cover WAL from a history the server abandoned. Incremental backups trust those summaries; a relation truncation recorded there discards the references to blocks modified before the switch point, and the restored blocks silently come from the prior backup. Summarize only up to the replay position during recovery. Backups taken on a standby start at a restartpoint, which is never ahead of replay, so they need no summaries beyond it. With this, a standby promoted at a recovery target no longer writes old-timeline summaries that cross the switch point. Such a summary can still be on disk from before a minor update, and the check added by the previous commit still refuses it. The summarizer is no longer ahead of replay at promotion either, so the check added by "Fix assertion failure in WAL summarizer after promotion" becomes defensive, and t/004_promote_past_target.pl now promotes while the summarizer waits at replay. t/014_summary_past_switch.pl now checks that the summarizer stops at replay and that the incremental backups restore the promoted server's rows; t/015_truncation_after_target.pl, from shihao zhong's report, uses a recovery target that is not at a checkpoint's redo pointer. Reported-by: shihao zhong <[email protected]> Co-authored-by: shihao zhong <[email protected]> Tested-by: Sai Pranav <[email protected]> Discussion: https://postgr.es/m/cahgqgwfbpzr2wkrku9j8w6e+-rzd6ovazs8n_kxvzfccfnj...@mail.gmail.com --- src/backend/backup/basebackup_incremental.c | 14 ++-- src/backend/postmaster/walsummarizer.c | 35 +++----- src/bin/pg_combinebackup/meson.build | 1 + .../t/014_summary_past_switch.pl | 54 +++++++----- .../t/015_truncation_after_target.pl | 82 +++++++++++++++++++ .../t/004_promote_past_target.pl | 51 ++++++++---- 6 files changed, 172 insertions(+), 65 deletions(-) create mode 100644 src/bin/pg_combinebackup/t/015_truncation_after_target.pl diff --git a/src/backend/backup/basebackup_incremental.c b/src/backend/backup/basebackup_incremental.c index 50b5fcf5ff..6958738411 100644 --- a/src/backend/backup/basebackup_incremental.c +++ b/src/backend/backup/basebackup_incremental.c @@ -570,12 +570,14 @@ PrepareForIncrementalBackup(IncrementalBackupInfo *ib, /* * A summary that ends after the point where we switched away from * this timeline also covers WAL that this server may never have - * replayed: a standby summarizes the WAL it has received, ahead of - * replay, and recovery can stop at a recovery target before the end - * of it. Block references from that WAL only make us copy more than - * needed, but a relation truncation recorded there would discard the - * references to blocks modified before the switch. Remember these - * summaries, so that the merge below can refuse such a truncation. + * replayed. A standby now summarizes only replayed WAL, but a + * summary written by an older release, which followed the + * walreceiver ahead of replay, may still be on disk, and recovery can + * have stopped at a recovery target before the end of it. Block + * references from that WAL only make us copy more than needed, but a + * relation truncation recorded there would discard the references to + * blocks modified before the switch. Remember these summaries, so + * that the merge below can refuse such a truncation. */ if (tle->tli != backup_state->starttli) { diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 88b6fee82b..a065bdb5b5 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -846,10 +846,6 @@ GetLatestLSN(TimeLineID *tli) } else { - XLogRecPtr flush_lsn; - TimeLineID flush_tli; - XLogRecPtr replay_lsn; - TimeLineID replay_tli; TimeLineID insert_tli; /* @@ -868,26 +864,19 @@ GetLatestLSN(TimeLineID *tli) } /* - * What we really want to know is how much WAL has been flushed to - * disk, but the only flush position available is the one provided by - * the walreceiver, which may not be running, because this could be - * crash recovery or recovery via restore_command. So use either the - * WAL receiver's flush position or the replay position, whichever is - * further ahead, on the theory that if the WAL has been replayed then - * it must also have been flushed to disk. + * Otherwise, summarize only WAL that has been replayed. A walreceiver + * may have flushed WAL beyond that point, but it is not yet part of + * this server's history: recovery may stop at a recovery target before + * reaching it, and if the server is then promoted, that WAL belongs to + * a timeline this server abandoned. A summary covering it would + * describe changes, such as relation truncations, that never happened + * here, and incremental backups would trust it. + * + * Backups taken during recovery start at a restartpoint, which is + * never ahead of replay, so they don't need summaries of anything + * further. */ - flush_lsn = GetWalRcvFlushRecPtr(NULL, &flush_tli); - replay_lsn = GetXLogReplayRecPtr(&replay_tli); - if (flush_lsn > replay_lsn) - { - *tli = flush_tli; - return flush_lsn; - } - else - { - *tli = replay_tli; - return replay_lsn; - } + return GetXLogReplayRecPtr(tli); } } diff --git a/src/bin/pg_combinebackup/meson.build b/src/bin/pg_combinebackup/meson.build index 1d5f35af06..d5fdd950c7 100644 --- a/src/bin/pg_combinebackup/meson.build +++ b/src/bin/pg_combinebackup/meson.build @@ -42,6 +42,7 @@ tests += { 't/012_vm_consistency.pl', 't/013_summary_order.pl', 't/014_summary_past_switch.pl', + 't/015_truncation_after_target.pl', ], } } diff --git a/src/bin/pg_combinebackup/t/014_summary_past_switch.pl b/src/bin/pg_combinebackup/t/014_summary_past_switch.pl index b1c0bf8c17..5af1a85e5e 100644 --- a/src/bin/pg_combinebackup/t/014_summary_past_switch.pl +++ b/src/bin/pg_combinebackup/t/014_summary_past_switch.pl @@ -1,10 +1,10 @@ # Copyright (c) 2026, PostgreSQL Global Development Group # -# A streaming standby summarizes the WAL it receives, ahead of replay. When -# it is promoted at a recovery target inside a WAL summary, the old-timeline -# summary covering the target also covers WAL that the promoted server never -# replayed. Block references from that WAL are harmless, but a truncation -# recorded there must not be applied to an incremental backup. +# A streaming standby promoted at a recovery target inside a WAL summary, +# after receiving WAL past the target, including a truncation. A standby only +# summarizes replayed WAL, so the old-timeline summaries end at the switch +# point, and incremental backups after promotion restore the promoted +# server's rows. use strict; use warnings FATAL => 'all'; @@ -57,8 +57,8 @@ $primary->safe_psql('postgres', 'CHECKPOINT'); my $end_lsn = $primary->safe_psql('postgres', 'SELECT redo_lsn FROM pg_control_checkpoint()'); -# Start a standby that pauses at the target and has summarized everything -# the primary wrote, then promote it. +# Start a standby that pauses at the target and has received everything the +# primary wrote, then promote it. sub promoted_standby { my ($name, $target) = @_; @@ -73,11 +73,17 @@ EOF $node->poll_query_until('postgres', "SELECT pg_get_wal_replay_pause_state() = 'paused'") or die "timed out waiting for $name to pause"; - $node->poll_query_until('postgres', <<EOF) -SELECT EXISTS (SELECT FROM pg_available_wal_summaries() - WHERE tli = 1 AND end_lsn >= '$end_lsn') -EOF - or die "timed out waiting for $name to summarize up to $end_lsn"; + $node->poll_query_until('postgres', + "SELECT pg_last_wal_receive_lsn() >= '$end_lsn'") + or die "timed out waiting for $name to receive up to $end_lsn"; + $node->poll_query_until('postgres', + "SELECT pending_lsn >= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()") + or die "timed out waiting for $name to summarize up to replay"; + is( $node->safe_psql( + 'postgres', + "SELECT pending_lsn <= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()"), + 't', + "$name: the summarizer does not read past replay"); $node->promote; return $node; } @@ -90,8 +96,7 @@ $primary->stop; # hint bits would WAL-log their blocks again on the new timeline. my $contents_sql = "SELECT id, right(pad, 1) FROM t ORDER BY id"; -# First standby: the summary past the switch only adds block references, so -# the incremental backup works and restores the promoted server's rows. +# First standby: only new rows follow its target. $standby1->command_ok( [ 'pg_basebackup', '--no-sync', @@ -114,17 +119,28 @@ is($restored->safe_psql('postgres', $contents_sql), $expected1, 'restored rows match the first standby'); $restored->stop; -# Second standby: the summary past the switch records the truncation, which -# never happened on this server, so the incremental backup must refuse it. -$standby2->command_fails_like( +# Second standby: the truncation after its target was received but never +# replayed or summarized, so the incremental backup restores the rows. +$standby2->command_ok( [ 'pg_basebackup', '--no-sync', '--pgdata' => $primary->backup_dir . '/incr2', '--checkpoint' => 'fast', '--incremental' => "$full_path/backup_manifest", ], - qr/cannot use WAL summary file .* for an incremental backup/, - 'incremental backup refuses a truncation past the switch point'); + 'incremental backup when a truncation follows the target'); +my $expected2 = "1|x\n2|x\n3|o\n4|o\n5|x"; +is($standby2->safe_psql('postgres', $contents_sql), + $expected2, 'second standby has the rows replayed before the target'); $standby2->stop; +my $restored2 = PostgreSQL::Test::Cluster->new('restored2'); +$restored2->init_from_backup( + $primary, 'incr2', + combine_with_prior => ['full'], + combine_mode => $mode); +$restored2->start; +is($restored2->safe_psql('postgres', $contents_sql), + $expected2, 'restored rows match the second standby'); +$restored2->stop; done_testing(); diff --git a/src/bin/pg_combinebackup/t/015_truncation_after_target.pl b/src/bin/pg_combinebackup/t/015_truncation_after_target.pl new file mode 100644 index 0000000000..c30a7e8819 --- /dev/null +++ b/src/bin/pg_combinebackup/t/015_truncation_after_target.pl @@ -0,0 +1,82 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group +# +# A standby can summarize WAL past its recovery target. If that WAL +# truncates a table, an incremental backup taken after promotion must still +# include blocks modified before the target. + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 1); +$primary->append_conf('postgresql.conf', <<EOF); +summarize_wal = on +autovacuum = off +EOF +$primary->start; + +# One row per block. +my $pad_size = $primary->safe_psql('postgres', 'SHOW block_size') / 2; +$primary->safe_psql('postgres', <<EOF); +CREATE TABLE t (id int, v int, pad text STORAGE PLAIN); +INSERT INTO t SELECT g, 0, repeat('x', $pad_size) FROM generate_series(1, 4) g; +VACUUM t; +EOF + +$primary->backup('full'); +my $full_path = $primary->backup_dir . '/full'; + +my $standby = PostgreSQL::Test::Cluster->new('standby'); +$standby->init_from_backup($primary, 'full', has_streaming => 1); +$standby->append_conf('postgresql.conf', <<EOF); +summarize_wal = on +recovery_target_action = 'pause' +EOF + +# Modify the last block, and stop recovery right after that. +$primary->safe_psql('postgres', 'UPDATE t SET v = 1 WHERE id = 4'); +my $target = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()'); +$standby->append_conf('postgresql.conf', "recovery_target_lsn = '$target'\n"); +$standby->start; + +# Truncate the table on the old timeline only. +$primary->safe_psql('postgres', 'DELETE FROM t WHERE id > 1'); +$primary->safe_psql('postgres', 'VACUUM t'); +$primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); +$primary->safe_psql('postgres', 'CHECKPOINT'); + +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "standby did not pause at the target"; + +# Give the summarizer time to get past the target, if it is going to. +sleep(5); + +$primary->stop; +$standby->safe_psql('postgres', 'SELECT pg_promote()'); +$standby->poll_query_until('postgres', 'SELECT NOT pg_is_in_recovery()') + or die "standby was not promoted"; + +# Back up before reading t, so that setting hint bits can't hide the problem. +$standby->command_ok( + [ + 'pg_basebackup', '--no-sync', + '--pgdata' => $primary->backup_dir . '/incr', + '--checkpoint' => 'fast', + '--incremental' => "$full_path/backup_manifest", + ], + 'incremental backup after promotion'); +$standby->stop; + +my $restored = PostgreSQL::Test::Cluster->new('restored'); +$restored->init_from_backup($primary, 'incr', combine_with_prior => ['full']); +$restored->start; +is($restored->safe_psql('postgres', 'SELECT id, v FROM t ORDER BY id'), + "1|0\n2|0\n3|0\n4|1", 'restored backup has the update'); +$restored->stop; + +done_testing(); diff --git a/src/bin/pg_walsummary/t/004_promote_past_target.pl b/src/bin/pg_walsummary/t/004_promote_past_target.pl index 7672183c73..87314881cd 100644 --- a/src/bin/pg_walsummary/t/004_promote_past_target.pl +++ b/src/bin/pg_walsummary/t/004_promote_past_target.pl @@ -1,11 +1,13 @@ # Copyright (c) 2021-2026, PostgreSQL Global Development Group # -# A standby that stops at a recovery target can have summarized WAL beyond -# that target, because the WAL summarizer reads whatever the walreceiver has -# flushed. When such a standby is promoted, the old timeline ends at a point -# the summarizer has already passed. The summarizer must back up to the switch -# point and continue on the new timeline, rather than report a summary that -# ends before it started (which used to trip an assertion). +# Promote a standby that stopped at a recovery target while its WAL +# summarizer was waiting for more WAL. The summarizer must notice that the +# old timeline became historic and continue on the new timeline from the +# switch point. A standby summarizes only replayed WAL, so the summarizer +# waits at the switch point here, although the standby has received WAL past +# it. When a standby summarized what the walreceiver had flushed, the switch +# point could precede the start of the summary being written, which used to +# trip an assertion. use strict; use warnings FATAL => 'all'; @@ -24,13 +26,14 @@ $primary->start; $primary->safe_psql('postgres', 'CREATE TABLE t (i int)'); $primary->backup('backup'); -# The recovery target is the redo pointer of a checkpoint, so that the -# standby's summarizer completes a summary that starts exactly there before -# promotion; a second checkpoint lets it summarize past the target. +# The recovery target is the redo pointer of a checkpoint; the standby also +# receives a second checkpoint past it. $primary->safe_psql('postgres', 'INSERT INTO t VALUES (1); CHECKPOINT;'); my $target_lsn = $primary->safe_psql('postgres', 'SELECT redo_lsn FROM pg_control_checkpoint()'); $primary->safe_psql('postgres', 'INSERT INTO t VALUES (2); CHECKPOINT;'); +my $end_lsn = + $primary->safe_psql('postgres', 'SELECT pg_current_wal_flush_lsn()'); my $standby = PostgreSQL::Test::Cluster->new('standby'); $standby->init_from_backup($primary, 'backup', has_streaming => 1); @@ -42,33 +45,47 @@ log_min_messages = debug1 EOM $standby->start; -# Wait until replay is paused at the target, the summarizer has summarized -# past it on timeline 1, and the summarizer is waiting for more WAL. +# Wait until replay is paused at the target, the standby has received the +# WAL past it, and the summarizer has reached replay and is waiting for more +# WAL. $standby->poll_query_until('postgres', "SELECT pg_get_wal_replay_pause_state() = 'paused'") or die "timed out waiting for recovery to pause"; -$standby->poll_query_until('postgres', <<EOM) -SELECT EXISTS (SELECT FROM pg_available_wal_summaries() - WHERE tli = 1 AND start_lsn = '$target_lsn') -EOM - or die "timed out waiting for summary starting at $target_lsn"; +$standby->poll_query_until('postgres', + "SELECT pg_last_wal_receive_lsn() >= '$end_lsn'") + or die "timed out waiting for the standby to receive up to $end_lsn"; +$standby->poll_query_until('postgres', + "SELECT pending_lsn >= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()" +) or die "timed out waiting for summarization to reach replay"; $standby->poll_query_until('postgres', <<EOM) SELECT EXISTS (SELECT FROM pg_stat_activity WHERE backend_type = 'walsummarizer' AND wait_event = 'WalSummarizerWal') EOM or die "timed out waiting for summarizer to wait for WAL"; +is( $standby->safe_psql( + 'postgres', + "SELECT pending_lsn <= pg_last_wal_replay_lsn() FROM pg_get_wal_summarizer_state()" + ), + 't', + 'the summarizer does not read past replay'); my $log_offset = -s $standby->logfile; $standby->promote; $standby->safe_psql('postgres', 'CHECKPOINT'); -# Summarization must resume on timeline 2 exactly at the switch point. +# Summarization must end on timeline 1 and resume on timeline 2 exactly at +# the switch point. my $result = $standby->poll_query_until('postgres', <<EOM); SELECT EXISTS (SELECT FROM pg_available_wal_summaries() WHERE tli = 2 AND start_lsn = '$target_lsn') EOM ok($result, "summarization resumed on timeline 2 at $target_lsn"); +is( $standby->safe_psql( + 'postgres', + "SELECT max(end_lsn) FROM pg_available_wal_summaries() WHERE tli = 1"), + $target_lsn, + "timeline 1 summaries end at $target_lsn"); ok( !$standby->log_contains( qr/WAL summarizer process .* was terminated by signal/, $log_offset), -- 2.56.0
