On Fri, 13 Mar 2026 Greg Sabino Mullane <htamfids(at)gmail(dot)com> wrote:
> +1. I don't think we need to measure any times, but we do need to
> exercise that whole part of the code, ...
Added a case with a positive but small wal_sender_shutdown_timeout
to the test.
Regards,
Andrey Silitskiy
From b184e06208cee321d9fe581a5986748bcbbebea3 Mon Sep 17 00:00:00 2001
From: Alexander Korotkov <[email protected]>
Date: Wed, 11 Mar 2026 15:46:44 +0200
Subject: [PATCH v5] Introduce a new 'wal_sender_shutdown_timeout' GUC
Previously, at shutdown, walsender processes were always waiting to send all
pending data and ensure that all data was flushed to the remote node. But in
some cases, an unexpected wait may be unacceptable. For example, in logical
replication, apply_workers may hang on locks for some time, preventing the
sender from shutting down.
New guc allows specifying the maximum time the receiver can wait for a flush
of WAL data without changing the default behavior.
The value of -1 (default value) disables the timeout. If set, the walsender
will wait for all WALs to be flushed on the receiver side before exiting the
process.
If timeout is enabled, the walsender will exit after expiration without
confirming the remote flush. This may break the consistency between sender
and receiver. This timeout might be useful for a system with a high-latency
network (to reduce the time to shutdown) or to allow the publisher to
shutdown even when the subscribers' apply_worker is waiting for locks to be
released.
Discussion: https://postgr.es/m/TYAPR01MB586668E50FC2447AD7F92491F5E89%40TYAPR01MB5866.jpnprd01.prod.outlook.com
Author: Andrey Silitskiy <[email protected]>
Co-authored-by: Hayato Kuroda <[email protected]>
Reviewed-by: Ashutosh Bapat <[email protected]>
Reviewed-by: Kyotaro Horiguchi <[email protected]>
Reviewed-by: Amit Kapila <[email protected]>
Reviewed-by: Dilip Kumar <[email protected]>
Reviewed-by: Masahiko Sawada <[email protected]>
Reviewed-by: Andres Freund <[email protected]>
Reviewed-by: Takamichi Osumi <[email protected]>
Reviewed-by: Peter Smith <[email protected]>
Reviewed-by: Greg Sabino Mullane <[email protected]>
Reviewed-by: Vitaly Davydov <[email protected]>
Reviewed-by: Fujii Masao <[email protected]>
Reviewed-by: Ronan Dunklau <[email protected]>
Reviewed-by: Michael Paquier <[email protected]>
Reviewed-by: Japin Li <[email protected]>
---
doc/src/sgml/config.sgml | 34 +++
doc/src/sgml/high-availability.sgml | 9 +-
src/backend/replication/walsender.c | 117 +++++++++-
src/backend/utils/misc/guc_parameters.dat | 10 +
src/backend/utils/misc/postgresql.conf.sample | 4 +
src/include/replication/walsender.h | 1 +
src/test/subscription/meson.build | 1 +
.../t/038_walsnd_shutdown_timeout.pl | 209 ++++++++++++++++++
8 files changed, 379 insertions(+), 6 deletions(-)
create mode 100644 src/test/subscription/t/038_walsnd_shutdown_timeout.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8cdd826fbd3..8f8ef3a478c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4806,6 +4806,40 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-wal_sender_shutdown_timeout" xreflabel="wal_sender_shutdown_timeout">
+ <term><varname>wal_sender_shutdown_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>wal_sender_shutdown_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum period of time the walsender process will wait
+ for a successful flush of WAL data by the receiver after receipt of
+ a shutdown request. If this value is specified without units, it is
+ taken as milliseconds. A value of -1 disables the timeout mechanism.
+ It is disabled by default. This parameter can be set individually
+ for each walsender.
+ </para>
+ <para>
+ If disabled, the walsender will wait for all WAL data to be
+ successfully flushed on the receiver side before exiting the process.
+ This helps to keep the sender and receiver in sync after shutdown,
+ which is especially important for physical replication switchovers.
+ However, it can delay server shutdown.
+ </para>
+ <para>
+ If enabled, then after receipt of a shutdown request, the walsender
+ process will wait for the specified timeout and terminate without
+ further waiting for WAL data replication to the receiver. This can
+ reduce shutdown time when flushing WAL data to the receiver would
+ take a long time, for example, on high-latency networks or when
+ the subscriber's apply worker is blocked waiting for locks in
+ logical replication.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index c3f269e0364..2f67d6dfd18 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -1190,10 +1190,11 @@ primary_slot_name = 'node_a_slot'
</para>
<para>
- Users will stop waiting if a fast shutdown is requested. However, as
- when using asynchronous replication, the server will not fully
- shutdown until all outstanding WAL records are transferred to the currently
- connected standby servers.
+ Users will stop waiting if a fast shutdown is requested. However, if
+ <varname>wal_sender_shutdown_timeout</varname> is not set, the server will
+ not fully shutdown until all outstanding WAL records are transferred to
+ the currently connected standby servers. This waiting applies to both
+ asynchronous and synchronous replication.
</para>
</sect3>
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 79fc192b171..238731e831c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -35,6 +35,8 @@
* checkpoint finishes, the postmaster sends us SIGUSR2. This instructs
* walsender to send any outstanding WAL, including the shutdown checkpoint
* record, wait for it to be replicated to the standby, and then exit.
+ * This waiting time can be limited by the wal_sender_shutdown_timeout
+ * parameter.
*
*
* Portions Copyright (c) 2010-2026, PostgreSQL Global Development Group
@@ -131,6 +133,11 @@ int max_wal_senders = 10; /* the maximum number of concurrent
* walsenders */
int wal_sender_timeout = 60 * 1000; /* maximum time to send one WAL
* data message */
+
+int wal_sender_shutdown_timeout = -1; /* maximum time to wait for
+ * flush by receiver after
+ * shutdown request */
+
bool log_replication_commands = false;
/*
@@ -190,6 +197,11 @@ static TimestampTz last_reply_timestamp = 0;
/* Have we sent a heartbeat message asking for reply, since last reply? */
static bool waiting_for_ping_response = false;
+/*
+ * Timestamp of receipt of shutdown request by walsender.
+ */
+static TimestampTz shutdown_request_timestamp = 0;
+
/*
* While streaming WAL in Copy mode, streamingDoneSending is set to true
* after we have sent CopyDone. We should not send any more CopyData messages
@@ -263,6 +275,7 @@ static void WalSndKill(int code, Datum arg);
pg_noreturn static void WalSndShutdown(void);
static void XLogSendPhysical(void);
static void XLogSendLogical(void);
+pg_noreturn static void WalSndDoneImmediate(void);
static void WalSndDone(WalSndSendDataCallback send_data);
static void IdentifySystem(void);
static void UploadManifest(void);
@@ -282,6 +295,7 @@ static void ProcessPendingWrites(void);
static void WalSndKeepalive(bool requestReply, XLogRecPtr writePtr);
static void WalSndKeepaliveIfNecessary(void);
static void WalSndCheckTimeOut(void);
+static void WalSndCheckShutdownTimeout(void);
static long WalSndComputeSleeptime(TimestampTz now);
static void WalSndWait(uint32 socket_events, long timeout, uint32 wait_event);
static void WalSndPrepareWrite(LogicalDecodingContext *ctx, XLogRecPtr lsn, TransactionId xid, bool last_write);
@@ -1678,6 +1692,9 @@ ProcessPendingWrites(void)
/* Try to flush pending output to the client */
if (pq_flush_if_writable() != 0)
WalSndShutdown();
+
+ /* If wal_sender_shutdown_timeout is expired, exit the process */
+ WalSndCheckShutdownTimeout();
}
/* reactivate latch so WalSndLoop knows to continue */
@@ -2824,12 +2841,11 @@ ProcessStandbyPSRequestMessage(void)
static long
WalSndComputeSleeptime(TimestampTz now)
{
+ TimestampTz wakeup_time;
long sleeptime = 10000; /* 10 s */
if (wal_sender_timeout > 0 && last_reply_timestamp > 0)
{
- TimestampTz wakeup_time;
-
/*
* At the latest stop sleeping once wal_sender_timeout has been
* reached.
@@ -2850,6 +2866,20 @@ WalSndComputeSleeptime(TimestampTz now)
sleeptime = TimestampDifferenceMilliseconds(now, wakeup_time);
}
+ if (shutdown_request_timestamp != 0 && wal_sender_shutdown_timeout > 0)
+ {
+ long shutdown_sleeptime;
+
+ wakeup_time = TimestampTzPlusMilliseconds(shutdown_request_timestamp,
+ wal_sender_shutdown_timeout);
+
+ shutdown_sleeptime = TimestampDifferenceMilliseconds(now, wakeup_time);
+
+ /* Choose the earliest wakeup. */
+ if (shutdown_sleeptime < sleeptime)
+ sleeptime = shutdown_sleeptime;
+ }
+
return sleeptime;
}
@@ -2891,6 +2921,41 @@ WalSndCheckTimeOut(void)
}
}
+/*
+ * Check whether the walsender process should terminate due to the expiration
+ * of wal_sender_shutdown_timeout after the receipt of a shutdown request.
+ */
+static void
+WalSndCheckShutdownTimeout(void)
+{
+ TimestampTz now;
+
+ /* Ignored if wal_sender_shutdown_timeout is disabled. */
+ if (wal_sender_shutdown_timeout == -1)
+ return;
+
+ if (!(got_STOPPING || got_SIGUSR2))
+ return;
+
+ /* Terminate immediately if wal_sender_shutdown_timeout is set to 0. */
+ if (wal_sender_shutdown_timeout == 0)
+ WalSndDoneImmediate();
+
+ now = GetCurrentTimestamp();
+
+ if (shutdown_request_timestamp == 0)
+ {
+ shutdown_request_timestamp = now;
+ return;
+ }
+
+ if (TimestampDifferenceExceeds(shutdown_request_timestamp, now,
+ wal_sender_shutdown_timeout))
+ {
+ WalSndDoneImmediate();
+ }
+}
+
/* Main loop of walsender process that streams the WAL over Copy messages. */
static void
WalSndLoop(WalSndSendDataCallback send_data)
@@ -2945,6 +3010,12 @@ WalSndLoop(WalSndSendDataCallback send_data)
if (pq_flush_if_writable() != 0)
WalSndShutdown();
+ /*
+ * Check for wal_sender_shutdown_timeout. If timeout is expired, we do
+ * not wait for successful sending of all data to the receiver.
+ */
+ WalSndCheckShutdownTimeout();
+
/* If nothing remains to be sent right now ... */
if (WalSndCaughtUp && !pq_is_send_pending())
{
@@ -3593,6 +3664,48 @@ XLogSendLogical(void)
}
}
+/*
+ * Forced shutdown of walsender if wal_sender_shutdown_timeout has expired.
+ */
+static void
+WalSndDoneImmediate(void)
+{
+ WalSndState state = MyWalSnd->state;
+
+ if (state == WALSNDSTATE_CATCHUP ||
+ state == WALSNDSTATE_STREAMING ||
+ state == WALSNDSTATE_STOPPING)
+ {
+ QueryCompletion qc;
+
+ /* Try to inform receiver that XLOG streaming is done */
+ SetQueryCompletion(&qc, CMDTAG_COPY, 0);
+ EndCommand(&qc, DestRemote, false);
+
+ /*
+ * Note that the output buffer may be full during the forced shutdown of
+ * walsender. If pq_flush() is called at that time, the walsender process
+ * will be stuck. Therefore, call pq_flush_if_writable() instead.
+ * Successful reception of the done message with the walsender forced into
+ * a shutdown is not guaranteed.
+ */
+ pq_flush_if_writable();
+ }
+
+ /*
+ * Prevent ereport from attempting to send any more messages to the
+ * standby. Otherwise, it can cause the process to get stuck if the output
+ * buffers are full.
+ */
+ if (whereToSendOutput == DestRemote)
+ whereToSendOutput = DestNone;
+
+ ereport(WARNING,
+ (errmsg("terminating walsender due to wal_sender_shutdown_timeout expiration, replication may be incomplete")));
+
+ proc_exit(0);
+}
+
/*
* Shutdown if the sender is caught up.
*
diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat
index a5a0edf2534..8a28c7817c8 100644
--- a/src/backend/utils/misc/guc_parameters.dat
+++ b/src/backend/utils/misc/guc_parameters.dat
@@ -3448,6 +3448,16 @@
check_hook => 'check_wal_segment_size',
},
+{ name => 'wal_sender_shutdown_timeout', type => 'int', context => 'PGC_USERSET', group => 'REPLICATION_SENDING',
+ short_desc => 'Sets the maximum time to wait for receiver to flush WAL data after shutdown request.',
+ long_desc => '-1 disables timeout; 0 means immediate termination of walsender',
+ flags => 'GUC_UNIT_MS',
+ variable => 'wal_sender_shutdown_timeout',
+ boot_val => '-1',
+ min => '-1',
+ max => 'INT_MAX',
+},
+
{ name => 'wal_sender_timeout', type => 'int', context => 'PGC_USERSET', group => 'REPLICATION_SENDING',
short_desc => 'Sets the maximum time to wait for WAL replication.',
flags => 'GUC_UNIT_MS',
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e686d88afc4..7a6a006baff 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -349,6 +349,10 @@
#max_slot_wal_keep_size = -1 # in megabytes; -1 disables
#idle_replication_slot_timeout = 0 # in seconds; 0 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
+#wal_sender_shutdown_timeout = -1 # max time to wait for receiver to flush data
+ # after receipt of shutdown request; in milliseconds
+ # -1 disables (means waiting for complete flush)
+ # 0 means immediate termination of walsender
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
diff --git a/src/include/replication/walsender.h b/src/include/replication/walsender.h
index a4df3b8e0ae..999876b7699 100644
--- a/src/include/replication/walsender.h
+++ b/src/include/replication/walsender.h
@@ -33,6 +33,7 @@ extern PGDLLIMPORT bool wake_wal_senders;
/* user-settable parameters */
extern PGDLLIMPORT int max_wal_senders;
extern PGDLLIMPORT int wal_sender_timeout;
+extern PGDLLIMPORT int wal_sender_shutdown_timeout;
extern PGDLLIMPORT bool log_replication_commands;
extern void InitWalSender(void);
diff --git a/src/test/subscription/meson.build b/src/test/subscription/meson.build
index f4a9cf5057f..e71e95c6297 100644
--- a/src/test/subscription/meson.build
+++ b/src/test/subscription/meson.build
@@ -47,6 +47,7 @@ tests += {
't/035_conflicts.pl',
't/036_sequences.pl',
't/037_except.pl',
+ 't/038_walsnd_shutdown_timeout.pl',
't/100_bugs.pl',
],
},
diff --git a/src/test/subscription/t/038_walsnd_shutdown_timeout.pl b/src/test/subscription/t/038_walsnd_shutdown_timeout.pl
new file mode 100644
index 00000000000..120006a7800
--- /dev/null
+++ b/src/test/subscription/t/038_walsnd_shutdown_timeout.pl
@@ -0,0 +1,209 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Checks that the publisher is able to shut down without
+# waiting for sending of all pending data to the subscriber
+# with wal_sender_shutdown_timeout set
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+sub test_shutdown_with_empty_buffers
+{
+ my ($publisher, $subscriber, $bpgsql, $desc) = @_;
+
+ # start transaction on subscriber to hold locks
+ $bpgsql->query_safe("BEGIN; INSERT INTO pub_test VALUES (0);");
+
+ # run concurrent transaction on publisher and commit
+ $publisher->safe_psql('postgres',
+ 'BEGIN; INSERT INTO pub_test VALUES (0); COMMIT;');
+
+ my $log_offset = -s $publisher->logfile;
+
+ # test publisher shutdown
+ $publisher->stop('fast');
+ pass($desc);
+
+ ok( $publisher->log_contains(
+ qr/WARNING: .* terminating walsender due to wal_sender_shutdown_timeout expiration, replication may be incomplete/,
+ $log_offset),
+ "warning after walsender termination was emitted for: $desc");
+
+ return;
+}
+
+sub test_shutdown_with_full_buffers
+{
+ my ($publisher, $subscriber, $bpgsql, $desc) = @_;
+
+ # lock table to make apply_worker hang
+ $bpgsql->query_safe("BEGIN; LOCK TABLE pub_test IN EXCLUSIVE MODE;");
+
+ my $last_sent_lsn = $publisher->safe_psql('postgres',
+ "select sent_lsn from pg_stat_replication where application_name = 'sub_all';"
+ );
+ my $cur_sent_lsn;
+
+ # generate big amount of wal records for locked table
+ $publisher->safe_psql('postgres',
+ 'BEGIN; INSERT INTO pub_test SELECT i from generate_series(1, 20000) s(i); COMMIT;'
+ );
+
+ # wait for walsender to fill output buffers
+ my $max_attempts = $PostgreSQL::Test::Utils::timeout_default;
+ while ($max_attempts-- >= 0)
+ {
+ sleep 1;
+
+ $cur_sent_lsn = $publisher->safe_psql('postgres',
+ "select sent_lsn from pg_stat_replication where application_name = 'sub_all';"
+ );
+
+ my $diff = $publisher->safe_psql(
+ 'postgres', qq(
+ SELECT pg_wal_lsn_diff('$cur_sent_lsn', '$last_sent_lsn');
+ ));
+
+ last if $diff == 0;
+
+ $last_sent_lsn = $cur_sent_lsn;
+ }
+
+ my $log_offset = -s $publisher->logfile;
+
+ # test publisher shutdown
+ $publisher->stop('fast');
+ pass($desc);
+
+ ok( $publisher->log_contains(
+ qr/WARNING: .* terminating walsender due to wal_sender_shutdown_timeout expiration, replication may be incomplete/,
+ $log_offset),
+ "warning after walsender termination with full buffers was emitted for: $desc");
+
+ return;
+}
+
+sub cleanup_after_test_case
+{
+ my ($publisher, $bpgsql) = @_;
+
+ $bpgsql->query_safe("ABORT;");
+
+ $publisher->start();
+ $publisher->wait_for_catchup('sub_all');
+}
+
+# =============================================================================
+# Setup publisher and subscriber
+
+# create publisher
+my $publisher = PostgreSQL::Test::Cluster->new('publisher');
+$publisher->init(allows_streaming => 'logical');
+# set wal_sender_shutdown_timeout GUC parameter to immediate termination
+$publisher->append_conf(
+ 'postgresql.conf',
+ "wal_sender_timeout = 0
+ wal_sender_shutdown_timeout = 0");
+$publisher->start();
+
+# create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('subscriber');
+$subscriber->init();
+$subscriber->append_conf('postgresql.conf',
+ "wal_receiver_status_interval = 1");
+$subscriber->start();
+
+# create publication for test table
+$publisher->safe_psql(
+ 'postgres', q{
+ CREATE TABLE pub_test (id int PRIMARY KEY);
+ CREATE PUBLICATION pub_all FOR TABLE pub_test;
+});
+
+# create matching table on subscriber
+$subscriber->safe_psql(
+ 'postgres', q{
+ CREATE TABLE pub_test (id int PRIMARY KEY);
+});
+
+# form connection string to publisher
+my $pub_connstr = $publisher->connstr;
+
+# create the subscription on subscriber
+$subscriber->safe_psql(
+ 'postgres', qq{
+ CREATE SUBSCRIPTION sub_all
+ CONNECTION '$pub_connstr'
+ PUBLICATION pub_all;
+});
+
+# wait for initial sync to finish
+$subscriber->wait_for_subscription_sync($publisher, 'sub_all');
+
+# create background psql session
+my $bpgsql = $subscriber->background_psql('postgres', on_error_stop => 0);
+
+# =============================================================================
+
+# =============================================================================
+# Testcase: Shutdown of publisher when output buffers are not full
+# (wal_sender_shutdown_timeout = 0)
+
+test_shutdown_with_empty_buffers($publisher, $subscriber, $bpgsql,
+ 'successful shutdown of publisher when output buffers are not full (wal_sender_shutdown_timeout = 0)');
+
+# =============================================================================
+
+cleanup_after_test_case($publisher, $bpgsql);
+
+# =============================================================================
+# Testcase: Shutdown of publisher with full output buffers
+# (wal_sender_shutdown_timeout = 0)
+
+test_shutdown_with_full_buffers($publisher, $subscriber, $bpgsql,
+ 'successful shutdown of publisher with full output buffers (wal_sender_shutdown_timeout = 0)');
+
+# =============================================================================
+
+# =============================================================================
+# Change wal_sender_shutdown_timeout from 0 to 10ms
+
+$publisher->append_conf(
+ 'postgresql.conf',
+ "wal_sender_shutdown_timeout = 10ms");
+
+cleanup_after_test_case($publisher, $bpgsql);
+
+$publisher->safe_psql('postgres', q{
+ TRUNCATE pub_test;
+});
+
+$publisher->wait_for_catchup('sub_all');
+
+# =============================================================================
+
+# =============================================================================
+# Testcase: Shutdown of publisher when output buffers are not full
+# (wal_sender_shutdown_timeout = 10ms)
+
+test_shutdown_with_empty_buffers($publisher, $subscriber, $bpgsql,
+ 'successful shutdown of publisher when output buffers are not full (wal_sender_shutdown_timeout = 10ms)');
+
+# =============================================================================
+
+cleanup_after_test_case($publisher, $bpgsql);
+
+# =============================================================================
+# Testcase: Shutdown of publisher with full output buffers
+# (wal_sender_shutdown_timeout = 10ms)
+
+test_shutdown_with_full_buffers($publisher, $subscriber, $bpgsql,
+ 'successful shutdown of publisher with full output buffers (wal_sender_shutdown_timeout = 10ms)');
+
+# =============================================================================
+
+done_testing();
--
2.34.1
