On Wed, Nov 18, 2026 at 4:54 AM Zsolt Parragi <[email protected]> wrote:
> Hello > > 1. > > > Subject: [PATCH v2 2/2] Add TAP test for primary shutdown during > streaming > > This should also have a meson.build edit so the meson builds also run > the new test > > 2. Stopping the primary and waiting a bit gives this on the standby: > > LOG: replication terminated by primary server > DETAIL: End of WAL reached on timeline 1 at 0/030000B0. > FATAL: could not receive database system identifier and timeline ID > from the primary server: no connection to the server > > After the early return walreceiver waits for new orders and then runs > IDENTIFY_SYSTEM on the closed connection. The test passes because it > only checks that the old message is gone. > > 3. On master, a clean publisher shutdown hits the same error in the > logical apply worker, and that disables a disable_on_error > subscription: > > LOG: data stream from publisher has ended > ERROR: could not send end-of-streaming message to primary: server > closed the connection unexpectedly > LOG: subscription "sub" has been disabled because of an error > > With the patch only the first line remains and the subscription stays > enabled. This doesn't seem like a bad thing, but probably should be > mentioned at least in the commit message, and maybe tested too? > > 4. Partial WAL segment archived on archive_mode=always standbys > > Since endstreaming no longer fails, walreceiver continues into the > end-of-timeline block in WalReceiverMain(), which closes the current > segment and calls XLogArchiveNotify() on it, even though the segment > is incomplete. > > Setup: > > * standby with archive_mode = always and archive_command = 'test ! -f > arch/%f && cp %p arch/%f' > * stop the primary > * wait a few seconds > * start it again > * run pg_switch_wal() a few times: > > master: archived while primary down: (none) > pg_stat_archiver: 5|000000010000000000000007|0| > patched: archived while primary down: 000000010000000000000003 > pg_stat_archiver: > 1|000000010000000000000003|6|000000010000000000000003 > > 5. Would something like this work instead? > > libpqrcv_endstreaming(): > + *next_tli = 0; > + if (PQstatus(conn->streamConn) == CONNECTION_BAD) > + return; > walreceiver.c, after walrcv_endstreaming(wrconn, &primaryTLI): > + if (primaryTLI == 0) > + proc_exit(0); > Hi Zsolt, Thanks for the careful review. Agreed on points 1 to 4. I missed those follow ons in v2. On point 5 I like the direction, but CONNECTION_BAD alone does not cover the original report. After COPY 0, libpq has left COPY while PQstatus() can still be CONNECTION_OK, so PQputCopyEnd() still fails with no COPY in progress. That is why copyEnded stays. CONNECTION_BAD is a useful complement for an abrupt close, so I kept that too. v3 also does proc_exit(0) in physical walreceiver when endstreaming reports next_tli == 0. Timeline switch still sends a non zero next TLI and keeps waiting mode. Clean shutdown does not, and falling through was what caused the IDENTIFY_SYSTEM FATAL and the incomplete segment archive under archive_mode=always. Both TAP tests are in meson.build now, and they cover the endstreaming FATAL, IDENTIFY_SYSTEM, disable_on_error, incomplete segment archive, and reconnect after restart. Attached. Regards, Stepan Neretin
From 3ef58af9762947825974b560fae537bb46b04489 Mon Sep 17 00:00:00 2001 From: Stepan Neretin <[email protected]> Date: Wed, 18 Nov 2026 11:17:03 +0700 Subject: [PATCH v3 2/2] Add TAP tests for primary/publisher shutdown during streaming Verify that a clean remote shutdown does not FATAL on endstreaming, reuse the dead connection, archive an incomplete segment, or disable a disable_on_error subscription, and that streaming resumes after restart. Discussion: https://postgr.es/m/CA+Yyo5S_n3ZcZh-3LXNj2-rxqXnbGd8901Yres0LPUS=ttl...@mail.gmail.com --- src/test/recovery/meson.build | 1 + .../t/061_primary_shutdown_streaming.pl | 96 +++++++++++++++++++ src/test/subscription/meson.build | 1 + .../subscription/t/039_publisher_shutdown.pl | 71 ++++++++++++++ 4 files changed, 169 insertions(+) create mode 100644 src/test/recovery/t/061_primary_shutdown_streaming.pl create mode 100644 src/test/subscription/t/039_publisher_shutdown.pl diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b6ac9dd07a7..489413275d8 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -69,6 +69,7 @@ tests += { 't/058_shutdown_crash_restart.pl', 't/059_remote_apply_status_interval.pl', 't/060_multixact_crashed_updater.pl', + 't/061_primary_shutdown_streaming.pl', ], }, } diff --git a/src/test/recovery/t/061_primary_shutdown_streaming.pl b/src/test/recovery/t/061_primary_shutdown_streaming.pl new file mode 100644 index 00000000000..faf8de8f4b3 --- /dev/null +++ b/src/test/recovery/t/061_primary_shutdown_streaming.pl @@ -0,0 +1,96 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Clean primary shutdown must not FATAL in endstreaming, reuse the dead +# connection for IDENTIFY_SYSTEM, or archive an incomplete WAL segment +# under archive_mode=always. Streaming must resume after restart. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use Test::More; + +# pg_walfile_name() is unavailable during recovery. Assumes the default +# 16MB wal_segment_size used by the TAP suite. +sub walfile_from_lsn +{ + my ($lsn, $tli) = @_; + $tli = 1 unless defined $tli; + my ($hi_hex, $lo_hex) = split(m{/}, $lsn); + my $hi = hex($hi_hex); + my $lo = hex($lo_hex); + my $segsize = 16 * 1024 * 1024; + my $segs_per_id = 256; # 2^32 / 16MB + my $logSegNo = $hi * $segs_per_id + int($lo / $segsize); + return sprintf("%08X%08X%08X", $tli, int($logSegNo / $segs_per_id), + $logSegNo % $segs_per_id); +} + +my $node_primary = PostgreSQL::Test::Cluster->new('primary'); +$node_primary->init(allows_streaming => 1); +$node_primary->start; +$node_primary->backup('backup'); + +my $node_standby = PostgreSQL::Test::Cluster->new('standby'); +$node_standby->init_from_backup( + $node_primary, 'backup', + has_streaming => 1); +# init_from_backup ignores has_archiving. Set up standby archiving by hand. +$node_standby->enable_archiving; +$node_standby->append_conf('postgresql.conf', 'archive_mode = always'); +$node_standby->start; + +# Finish one segment, then leave the next one incomplete. +$node_primary->safe_psql('postgres', + "CREATE TABLE shutdown_stream_tab AS SELECT generate_series(1, 100) AS id"); +$node_primary->safe_psql('postgres', 'SELECT pg_switch_wal()'); +$node_primary->safe_psql('postgres', + "INSERT INTO shutdown_stream_tab VALUES (101)"); +$node_primary->wait_for_catchup($node_standby); + +my $log_offset = -s $node_standby->logfile; +$node_primary->stop; + +# Wait until a successor walreceiver fails to connect. That means the old +# process finished endstreaming and exited. +$node_standby->wait_for_log(qr/could not connect to the primary server/, + $log_offset); + +ok( !$node_standby->log_contains( + 'could not send end-of-streaming message', $log_offset), + 'walreceiver does not fail ending the stream after primary shutdown'); +ok( !$node_standby->log_contains( + 'could not receive database system identifier and timeline ID', + $log_offset), + 'walreceiver does not reuse the closed connection for IDENTIFY_SYSTEM'); + +# Shutdown checkpoint may complete the pre stop segment. Check the one that +# remains incomplete afterwards. +my $receive_lsn = $node_standby->safe_psql('postgres', + "SELECT pg_last_wal_receive_lsn()"); +my $incomplete_seg = walfile_from_lsn($receive_lsn); +my $archive_status = + $node_standby->data_dir . "/pg_wal/archive_status/$incomplete_seg.ready"; +ok( !-f $archive_status, + "incomplete segment $incomplete_seg does not get a .ready file"); +ok( !-f $node_standby->archive_dir . "/$incomplete_seg", + "incomplete segment $incomplete_seg is not archived while primary is down"); + +my $reconnect_offset = -s $node_standby->logfile; +$node_primary->start; +$node_standby->wait_for_log( + qr/(?:started streaming WAL|restarted WAL streaming)/, + $reconnect_offset); +$node_primary->safe_psql('postgres', + "INSERT INTO shutdown_stream_tab VALUES (102)"); +$node_primary->wait_for_catchup($node_standby); + +my $result = + $node_standby->safe_psql('postgres', + "SELECT count(*) FROM shutdown_stream_tab"); +is($result, '102', 'standby reconnects and catches up after primary restart'); + +$node_standby->stop; +$node_primary->stop; + +done_testing(); diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build index e71e95c6297..c4d790e40f0 100644 --- a/src/test/subscription/meson.build +++ b/src/test/subscription/meson.build @@ -48,6 +48,7 @@ tests += { 't/036_sequences.pl', 't/037_except.pl', 't/038_walsnd_shutdown_timeout.pl', + 't/039_publisher_shutdown.pl', 't/100_bugs.pl', ], }, diff --git a/src/test/subscription/t/039_publisher_shutdown.pl b/src/test/subscription/t/039_publisher_shutdown.pl new file mode 100644 index 00000000000..5daf49856f8 --- /dev/null +++ b/src/test/subscription/t/039_publisher_shutdown.pl @@ -0,0 +1,71 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Clean publisher shutdown must not ERROR in endstreaming or disable a +# disable_on_error subscription. Replication continues after restart. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use Test::More; + +my $node_publisher = PostgreSQL::Test::Cluster->new('publisher'); +$node_publisher->init(allows_streaming => 'logical'); +$node_publisher->start; + +my $node_subscriber = PostgreSQL::Test::Cluster->new('subscriber'); +$node_subscriber->init; +$node_subscriber->start; + +$node_publisher->safe_psql( + 'postgres', qq( + CREATE TABLE test_tab (id int PRIMARY KEY); + INSERT INTO test_tab VALUES (1); + CREATE PUBLICATION test_pub FOR TABLE test_tab; +)); + +my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres'; +$node_subscriber->safe_psql( + 'postgres', qq( + CREATE TABLE test_tab (id int PRIMARY KEY); + CREATE SUBSCRIPTION test_sub + CONNECTION '$publisher_connstr' + PUBLICATION test_pub + WITH (disable_on_error = true); +)); + +$node_subscriber->wait_for_subscription_sync($node_publisher, 'test_sub'); + +my $log_offset = -s $node_subscriber->logfile; +$node_publisher->stop; + +# Wait until a successor apply worker fails to connect. That means the +# previous process finished endstreaming with the subscription still enabled. +$node_subscriber->wait_for_log( + qr/apply worker for subscription "test_sub" could not connect to the publisher/, + $log_offset); + +ok( !$node_subscriber->log_contains( + 'could not send end-of-streaming message', $log_offset), + 'apply worker does not fail ending the stream after publisher shutdown'); +ok( !$node_subscriber->log_contains( + 'subscription "test_sub" has been disabled because of an error', + $log_offset), + 'disable_on_error subscription is not disabled on clean publisher shutdown'); + +my $enabled = $node_subscriber->safe_psql('postgres', + "SELECT subenabled FROM pg_subscription WHERE subname = 'test_sub'"); +is($enabled, 't', 'subscription remains enabled'); + +$node_publisher->start; +$node_publisher->safe_psql('postgres', "INSERT INTO test_tab VALUES (2)"); +$node_publisher->wait_for_catchup('test_sub'); + +my $result = $node_subscriber->safe_psql('postgres', + "SELECT count(*) FROM test_tab"); +is($result, '2', 'subscription resumes after publisher restart'); + +$node_subscriber->stop; +$node_publisher->stop; + +done_testing(); -- 2.56.0
From 0a20f28fb019020a9ed2e3186fb2373f85d8e0ef Mon Sep 17 00:00:00 2001 From: Stepan Neretin <[email protected]> Date: Wed, 18 Nov 2026 11:17:03 +0700 Subject: [PATCH v3 1/2] Fix FATAL when primary ends streaming with COPY 0 If the primary shuts down cleanly, walsender ends the COPY with CommandComplete ("COPY 0") and exits without waiting for CopyDone. libpqrcv_receive() already consumes that result, then libpqrcv_endstreaming() still calls PQputCopyEnd() and fails with "no COPY in progress", which becomes a FATAL in the walreceiver. Remember that CommandComplete already ended the COPY and skip PQputCopyEnd() in that case. Also skip it when the connection is already CONNECTION_BAD. Clear the flag in libpqrcv_startstreaming(), because receive() is also used for tablesync COPY TO STDOUT and that must not cause a later replication stream on the same connection to skip CopyDone. If endstreaming reports no next timeline, exit with proc_exit(0) instead of falling into waiting mode, so the dead connection is not reused and an incomplete segment is not archived. Discussion: https://postgr.es/m/CA+Yyo5S_n3ZcZh-3LXNj2-rxqXnbGd8901Yres0LPUS=ttl...@mail.gmail.com --- .../libpqwalreceiver/libpqwalreceiver.c | 19 +++++++++++++++++++ src/backend/replication/walreceiver.c | 11 +++++++++++ 2 files changed, 30 insertions(+) diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index dcc738f4e38..27eb2fd5a40 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -52,6 +52,8 @@ struct WalReceiverConn bool logical; /* Buffer for currently read records */ char *recvBuf; + /* set when CommandComplete already ended the COPY */ + bool copyEnded; }; /* Prototypes for interface functions */ @@ -618,6 +620,9 @@ libpqrcv_startstreaming(WalReceiverConn *conn, Assert(options->logical == conn->logical); Assert(options->slotname || !options->logical); + /* may be left set by a prior non streaming COPY */ + conn->copyEnded = false; + initStringInfo(&cmd); /* Build the command. */ @@ -709,6 +714,19 @@ libpqrcv_endstreaming(WalReceiverConn *conn, TimeLineID *next_tli) { PGresult *res; + /* + * Skip PQputCopyEnd() if CommandComplete already ended the COPY, or if + * the connection is already gone. copyEnded is still required after + * COPY 0 because PQstatus() may remain CONNECTION_OK. + */ + if (conn->copyEnded || + PQstatus(conn->streamConn) == CONNECTION_BAD) + { + conn->copyEnded = false; + *next_tli = 0; + return; + } + /* * Send copy-end message. As in libpqsrv_exec, this could theoretically * block, but the risk seems small. @@ -887,6 +905,7 @@ libpqrcv_receive(WalReceiverConn *conn, char **buffer, if (PQresultStatus(res) == PGRES_COMMAND_OK) { PQclear(res); + conn->copyEnded = true; /* Verify that there are no more results. */ res = libpqsrv_get_result(conn->streamConn, diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index 11b9a12c52c..28b96d9829f 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -29,6 +29,10 @@ * nudge the existing walreceiver process that's waiting, instead of launching * a new one. * + * A clean primary shutdown is different. There is no next timeline to follow + * on this connection, so walreceiver exits and a fresh process is started + * when streaming is requested again. + * * Normal termination is by SIGTERM, which instructs the walreceiver to * ereport(FATAL). Emergency termination is by SIGQUIT; like any postmaster * child process, the walreceiver will simply abort and exit on SIGQUIT. A @@ -691,6 +695,13 @@ WalReceiverMain(const void *startup_data, size_t startup_data_len) */ walrcv_endstreaming(wrconn, &primaryTLI); + /* + * Clean remote shutdown. Exit instead of waiting mode, which would + * reuse a dead connection and archive an incomplete segment. + */ + if (primaryTLI == 0) + proc_exit(0); + /* * If the server had switched to a new timeline that we didn't * know about when we began streaming, fetch its timeline history -- 2.56.0
