Hi Daria, Thanks for confirming the analysis independently, and for the review and testing.
On Thu, Oct 22, 2026 at 10:30 AM Daria Lepikhova <[email protected]> wrote: > 1. The comment in summarizer_read_local_xlog_page() about read_upto > moving backwards says the next iteration will "likely conclude that > we've reached end of WAL". It might be worth mentioning there that > SummarizeWAL() then reports no progress, so the link between the two > places is visible from both sides. Done in the attached v2. The comment now ends with "If the switch point precedes the start of the current summary, SummarizeWAL() then reports no progress." The comment on the clamp in SummarizeWAL() already points back to summarizer_read_local_xlog_page(). > 2. In the test, the summarizer PID check and the log check both detect > the same thing, a restart after the crash. Without the fix neither > is reached anyway: pg_ctl promote fails once the postmaster > reinitializes, and the test dies without a plan. I think one of the > two checks would be sufficient. Agreed. v2 keeps only the log check, since log_contains() doesn't need a running server. TAP nodes run with restart_after_crash = off, so the crash shuts the standby down instead of reinitializing it, 73-105 ms after it reaches production in my runs. pg_ctl promote polls every 100 ms, so whether it sees the promotion first is a race. Here promote won in 13 of 13 runs without the fix, and the v1 test then failed "not ok 1" and died on the PID query against the stopped server. With v2 and the fix reverted, it now ends with "failed 2 tests of 2" instead (4 of 4). When promote loses the race, as in your run, the test still bails out at promote. I left that as it is. v2 applies to master, 19, 18 and 17 like v1, and passes there. > Separately, while looking at this I found a related but different > problem: if the switch point falls exactly on a page or segment > boundary, [...] Looking forward to that thread. I'm happy to run your reproducer against v2 once you post it. Regards, Manu
From 0b208cb5f1cdd5226e6e7db6784baf12d20af1fb Mon Sep 17 00:00:00 2001 From: Manu <[email protected]> Date: Thu, 22 Oct 2026 11:15:44 -0300 Subject: [PATCH v2] Fix assertion failure in WAL summarizer after promotion A standby's WAL summarizer summarizes up to what the walreceiver has flushed, which can be well past the point where replay stops at a recovery target. If the standby is then promoted, the old timeline ends at a switch point that the summarizer has already passed. When the summarizer notices the new timeline while waiting for WAL inside SummarizeWAL(), summarizer_read_local_xlog_page() moves read_upto backwards to the switch point, as its comments anticipate, and the next read hits end of WAL. SummarizeWAL() then reported read_upto as the end of the summary, an LSN before the one it started from, and WalSummarizerMain() failed Assert(end_of_summary_lsn >= current_lsn) Builds without assertions were not affected in practice: the main loop treated the bogus end LSN as its new position, noticed the timeline switch on its next iteration and resumed from the switch point on the new timeline. Fix by having SummarizeWAL() report no progress in that case. The main loop then backs up to the switch point through the code that already exists for that purpose. Add a test reproducing the scenario reported by Fujii Masao. Reported-by: Fujii Masao <[email protected]> Reviewed-by: Daria Lepikhova <[email protected]> Tested-by: Daria Lepikhova <[email protected]> Discussion: https://postgr.es/m/cahgqgwfbpzr2wkrku9j8w6e+-rzd6ovazs8n_kxvzfccfnj...@mail.gmail.com Backpatch-through: 17 --- src/backend/postmaster/walsummarizer.c | 16 +++- src/bin/pg_walsummary/meson.build | 1 + .../t/004_promote_past_target.pl | 77 +++++++++++++++++++ 3 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 src/bin/pg_walsummary/t/004_promote_past_target.pl diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c index 8333a4e..88b6fee 100644 --- a/src/backend/postmaster/walsummarizer.c +++ b/src/backend/postmaster/walsummarizer.c @@ -1287,6 +1287,18 @@ SummarizeWAL(TimeLineID tli, XLogRecPtr start_lsn, bool exact, break; } + /* + * If the timeline we were reading became historic while we were waiting + * for more WAL, summarizer_read_local_xlog_page() may have moved read_upto + * backwards to a switch point that precedes the point where this summary + * started (for example, when a standby that has summarized WAL beyond its + * recovery target is promoted). In that case we have summarized nothing, + * so report that we made no progress. The caller will notice the timeline + * switch on its next iteration and back up to the switch point. + */ + if (summary_end_lsn < summary_start_lsn) + summary_end_lsn = summary_start_lsn; + /* Destroy xlogreader. */ pfree(xlogreader->private_data); XLogReaderFree(xlogreader); @@ -1729,7 +1741,9 @@ summarizer_read_local_xlog_page(XLogReaderState *state, * backwards, because we might have been promoted before * reaching the end of the previous timeline. In that * case, the next loop iteration will likely conclude that - * we've reached end of WAL. + * we've reached end of WAL. If the switch point precedes + * the start of the current summary, SummarizeWAL() then + * reports no progress. */ private_data->read_upto = switchpoint; diff --git a/src/bin/pg_walsummary/meson.build b/src/bin/pg_walsummary/meson.build index 839b644..46e0207 100644 --- a/src/bin/pg_walsummary/meson.build +++ b/src/bin/pg_walsummary/meson.build @@ -26,6 +26,7 @@ tests += { 't/001_basic.pl', 't/002_blocks.pl', 't/003_tli_switch.pl', + 't/004_promote_past_target.pl', ], } } diff --git a/src/bin/pg_walsummary/t/004_promote_past_target.pl b/src/bin/pg_walsummary/t/004_promote_past_target.pl new file mode 100644 index 0000000..7672183 --- /dev/null +++ b/src/bin/pg_walsummary/t/004_promote_past_target.pl @@ -0,0 +1,77 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group +# +# A standby that stops at a recovery target can have summarized WAL beyond +# that target, because the WAL summarizer reads whatever the walreceiver has +# flushed. When such a standby is promoted, the old timeline ends at a point +# the summarizer has already passed. The summarizer must back up to the switch +# point and continue on the new timeline, rather than report a summary that +# ends before it started (which used to trip an assertion). + +use strict; +use warnings FATAL => 'all'; +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 1); +$primary->append_conf('postgresql.conf', <<EOM); +autovacuum = off +summarize_wal = on +EOM +$primary->start; + +$primary->safe_psql('postgres', 'CREATE TABLE t (i int)'); +$primary->backup('backup'); + +# The recovery target is the redo pointer of a checkpoint, so that the +# standby's summarizer completes a summary that starts exactly there before +# promotion; a second checkpoint lets it summarize past the target. +$primary->safe_psql('postgres', 'INSERT INTO t VALUES (1); CHECKPOINT;'); +my $target_lsn = $primary->safe_psql('postgres', + 'SELECT redo_lsn FROM pg_control_checkpoint()'); +$primary->safe_psql('postgres', 'INSERT INTO t VALUES (2); CHECKPOINT;'); + +my $standby = PostgreSQL::Test::Cluster->new('standby'); +$standby->init_from_backup($primary, 'backup', has_streaming => 1); +$standby->append_conf('postgresql.conf', <<EOM); +recovery_target_lsn = '$target_lsn' +recovery_target_inclusive = false +recovery_target_action = 'pause' +log_min_messages = debug1 +EOM +$standby->start; + +# Wait until replay is paused at the target, the summarizer has summarized +# past it on timeline 1, and the summarizer is waiting for more WAL. +$standby->poll_query_until('postgres', + "SELECT pg_get_wal_replay_pause_state() = 'paused'") + or die "timed out waiting for recovery to pause"; +$standby->poll_query_until('postgres', <<EOM) +SELECT EXISTS (SELECT FROM pg_available_wal_summaries() + WHERE tli = 1 AND start_lsn = '$target_lsn') +EOM + or die "timed out waiting for summary starting at $target_lsn"; +$standby->poll_query_until('postgres', <<EOM) +SELECT EXISTS (SELECT FROM pg_stat_activity + WHERE backend_type = 'walsummarizer' + AND wait_event = 'WalSummarizerWal') +EOM + or die "timed out waiting for summarizer to wait for WAL"; + +my $log_offset = -s $standby->logfile; +$standby->promote; +$standby->safe_psql('postgres', 'CHECKPOINT'); + +# Summarization must resume on timeline 2 exactly at the switch point. +my $result = $standby->poll_query_until('postgres', <<EOM); +SELECT EXISTS (SELECT FROM pg_available_wal_summaries() + WHERE tli = 2 AND start_lsn = '$target_lsn') +EOM +ok($result, "summarization resumed on timeline 2 at $target_lsn"); + +ok( !$standby->log_contains( + qr/WAL summarizer process .* was terminated by signal/, $log_offset), + 'no WAL summarizer crash after promotion'); + +done_testing(); -- 2.56.0
