While I was working on my Latches vs Interrupts patch set, AI review
pointed out two race conditions that are present on master, too:
1. If a new catchup signal arrives while we're processing a previous
catchup signal, the new signal can be lost.
This is a pretty straightforward race condition.
ReceiveSharedInvalidMessages() clears catchupInterruptPending, after it
has processed all the messages. If another catchup interrupt arrives in
between, catchupInterruptPending is cleared even though there are now
more messages pending, and the "signaled" flag in shared memory is set
again.
This is relatively harmless: the consequence is that the sinval queue
might overflow, causing a cache reset, if nothing else wakes up the
process to read the invalidations. It's also very unlikely to occur in
practice; it requires the process to receive another catchup signal just
after it has caught up. That requires lots of new invalidations, and
even then, some other process is much more likely to be the laggard in
that situation.
But the fix is also pretty simple, see attached patch 0001.
2. Assertion failure if IDLE_STATS_UPDATE_TIMEOUT fires while processing
a catchup signal
I bumped into this while testing the first issue, and had a backend
paused in the middle of processing a catchup signal:
TRAP: failed Assert("s->state == TRANS_DEFAULT || s->state ==
TRANS_COMMIT || s->state == TRANS_ABORT || s->state == TRANS_PREPARE"),
File: "../master-wt/src/backend/access/transam/xact.c", Line: 902, PID:
23381
0 postgres ExceptionalCondition + 216
1 postgres GetCurrentTransactionStopTimestamp + 128
2 postgres pgstat_index_flush_cb + 204
3 postgres pgstat_flush_pending_entries + 248
4 postgres pgstat_report_stat + 436
5 postgres ProcessInterrupts + 3340
6 postgres errfinish + 840
7 postgres ReceiveSharedInvalidMessages + 572
8 postgres AcceptInvalidationMessages + 52
9 postgres AtStart_Cache + 28
10 postgres StartTransaction + 900
11 postgres StartTransactionCommand + 84
12 postgres ProcessCatchupInterrupt + 256
13 postgres ProcessClientReadInterrupt + 112
14 postgres secure_read + 456
15 postgres pq_recvbuf + 260
16 postgres pq_getbyte + 96
17 postgres SocketBackend + 44
18 postgres ReadCommand + 44
19 postgres PostgresMain + 2860
20 postgres BackendInitialize + 0
21 postgres postmaster_child_launch + 464
22 postgres BackendStartup + 304
23 postgres ServerLoop + 372
24 postgres PostmasterMain + 6560
25 postgres main + 924
In this particular case, CHECK_FOR_INTERRUPTS() was called from
errfinish(), which is only called because of a debugging message at
level DEBUG4, but there can be many other CHECK_FOR_INTERRUPTS() calls
in processing invalidations.
In ProcessInterrupts(), the condition for whether it's safe to process
the idle stats timeout looks like this:
/*
* If there are pending stats updates and we currently are truly idle
* (matching the conditions in PostgresMain(), report stats now.
*/
if (IdleStatsUpdateTimeoutPending &&
DoingCommandRead && !IsTransactionOrTransactionBlock())
{
IdleStatsUpdateTimeoutPending = false;
pgstat_report_stat(true);
}
Currently, IsTransactionOrTransactionBlock() returns false when we are
in the middle of StartTransaction(). That doesn't feel right to me, I
don't think a backend should be considered as "truly idle" when it's in
the middle of StartTransaction(). The attached patch 0002 changes
IsTransactionOrTransactionBlock() to return true (meaning not truly
idle) in that state.
Patch 0003 is a test that exercises both of those cases. It's AI
generated and I don't intend to commit it, I'm merely including it so
you can reproduce the failures if you want to. I don't like it because
it's pretty tightly coupled with injection points and the exact code
structure.
Any comments / objections on patches 0001 and 0002?
- Heikki
From 53d7d2f4d388e6d4df68d6fef60b25fbd7cc6be8 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Tue, 17 Nov 2026 00:18:14 +0200
Subject: [PATCH 1/3] Fix losing a sinval catchup signal that arrives during
catchup
If a new catchup signal arrives while we're still processing a
previous catchup signal, the new signal might be lost:
When a backend falls far behind in the shared invalidation queue,
SICleanupQueue() sets its "signaled" flag, and sends it
PROCSIG_CATCHUP_INTERRUPT, which sets the catchupInterruptPending flag
in the receiving backend. catchupInterruptPending is checked in the
receiving backend when it's idle, to process the catchup request. No
further catchup signals are sent to the backend while the "signaled"
flag is set. SIGetDataEntries() clears "signaled" when the backend
has caught up, after which it can be signaled again.
There was a race condition in how ReceiveSharedInvalidMessages() reset
catchupInterruptPending and "signaled". It cleared "signaled" first,
after consuming all the messages, and catchupInterruptPending only
after that. If a new catchup signal arrived after the "signaled" flag
was cleared in SIGetDataEntries(), but before clearing
catchupInterruptPending, the backend could be left with its "signaled"
flag set but catchupInterruptPending not set. As a result, other
backends would not send it any more catchup signals, but it would not
wake up to consume the messages until AcceptInvalidationMessages() was
called for some other reason, typically when it next ran a query.
To fix, reset catchupInterruptPending only if "signaled" was not
concurrently re-armed. Move the responsibility for resetting it into
SICleanupQueue() itself, because it already holds the lock needed to
read "signaled".
---
src/backend/storage/ipc/sinval.c | 16 +++++++++-------
src/backend/storage/ipc/sinvaladt.c | 16 ++++++++++++++++
2 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/src/backend/storage/ipc/sinval.c b/src/backend/storage/ipc/sinval.c
index 1540c7e0696..756776c8732 100644
--- a/src/backend/storage/ipc/sinval.c
+++ b/src/backend/storage/ipc/sinval.c
@@ -125,16 +125,18 @@ ReceiveSharedInvalidMessages(void (*invalFunction) (SharedInvalidationMessage *m
} while (nummsgs == MAXINVALMSGS);
/*
- * We are now caught up. If we received a catchup signal, reset that
- * flag, and call SICleanupQueue(). This is not so much because we need
- * to flush dead messages right now, as that we want to pass on the
- * catchup signal to the next slowest backend. "Daisy chaining" the
- * catchup signal this way avoids creating spikes in system load for what
- * should be just a background maintenance activity.
+ * We are now caught up. If we received a catchup signal, call
+ * SICleanupQueue(). This is not so much because we need to flush dead
+ * messages right now, as that we want to pass on the catchup signal to
+ * the next slowest backend. "Daisy chaining" the catchup signal this way
+ * avoids creating spikes in system load for what should be just a
+ * background maintenance activity.
+ *
+ * SICleanupQueue() resets catchupInterruptPending, unless we got signaled
+ * again in the meanwhile.
*/
if (catchupInterruptPending)
{
- catchupInterruptPending = false;
elog(DEBUG4, "sinval catchup complete, cleaning queue");
SICleanupQueue(false, 0);
}
diff --git a/src/backend/storage/ipc/sinvaladt.c b/src/backend/storage/ipc/sinvaladt.c
index 16880e7cc4a..4cc270ac629 100644
--- a/src/backend/storage/ipc/sinvaladt.c
+++ b/src/backend/storage/ipc/sinvaladt.c
@@ -556,6 +556,8 @@ SIGetDataEntries(SharedInvalidationMessage *data, int datasize)
* backends as "reset" in the array, and sending PROCSIG_CATCHUP_INTERRUPT
* to some backend that seems to be getting too far behind. We signal at
* most one backend at a time, for reasons explained at the top of the file.
+ * We also clear catchupInterruptPending, unless this backend has been
+ * signaled again since we last caught up.
*
* Caution: because we transiently release write lock when we have to signal
* some other backend, it is NOT guaranteed that there are still minFree
@@ -655,6 +657,20 @@ SICleanupQueue(bool callerHasWriteLock, int minFree)
else
segP->nextThreshold = (numMsgs / CLEANUP_QUANTUM + 1) * CLEANUP_QUANTUM;
+ /*
+ * Clear our local catchupInterruptPending flag, now that we have
+ * processed the catchup request.
+ *
+ * "signaled" is expected to be false if we were called in response to a
+ * catchup signal, because the caller consumes all the messages for this
+ * backend before performing cleanup. But it's possible that more
+ * messages arrived after that, causing us to be signaled again. Don't
+ * clear catchupInterruptPending in that case, or we'd miss the new
+ * catchup request.
+ */
+ if (!segP->procState[MyProcNumber].signaled)
+ catchupInterruptPending = false;
+
/*
* Lastly, signal anyone who needs a catchup interrupt. Since
* SendProcSignal() might not be fast, we don't want to hold locks while
--
2.47.3
From 9719223522fa3bf18adb5ba4342924dfac8fa18b Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Tue, 17 Nov 2026 00:18:16 +0200
Subject: [PATCH 2/3] Don't consider backend as fully idle when it's starting a
transaction
Make IsTransactionOrTransactionBlock() to return true, when we're in
the middle of StartTransaction(), i.e. when we're in the TRANS_START
state. This fixes an assertion failure in the following scenario:
1. Backend is completely idle, and receives a catchup interrupt.
ProcessCatchupInterrupt() calls StartTransactionCommand() to start
a new transaction
2. The "idle stats update" timeout occurs and sets
IdleStatsUpdateTimeoutPending.
3. In StartTransaction(), the sinval messages are processed when
AtStart_Cache() is called. If any CHECK_FOR_INTERRUPTS() calls are
made while processing the invalidations, it sees the pending
IdleStatsUpdateTimeoutPending flag and calls pgstat_report_stat()
4. pgstat_report_stat() calls GetCurrentTransactionStopTimestamp()
5. GetCurrentTransactionStopTimestamp() trips the assertion because
we're in TRANS_START state.
This commit fixes that scenario at step 3. ProcessInterrupts() checks
!IsTransactionOrTransactionBlock() to determine if the backend is
fully idle, and with this commit, it's not considered fully idle in
the TRANS_START state anymore.
Expand the comment in IsTransactionOrTransactionBlock() about when the
backend is considered fully idle.
---
src/backend/access/transam/xact.c | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 1227e3fbe24..1edda59ea32 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -5079,17 +5079,28 @@ IsTransactionBlock(void)
/*
* IsTransactionOrTransactionBlock --- are we within either a transaction
- * or a transaction block? (The backend is only really "idle" when this
- * returns false.)
+ * or a transaction block?
*
- * This should match up with IsTransactionBlock and IsTransactionState.
+ * The backend is only really "idle" when this returns false. Note that the
+ * purpose of this function is different from that of IsTransactionState().
+ * IsTransactionState() is used to check if we're in a live transaction where
+ * it's safe to access the database, while !IsTransactionOrTransactionBlock()
+ * is used to check if we're completely out of a transaction. When we're in
+ * the middle of starting or ending a transaction, we're neither in a live
+ * transaction nor completely out of a transaction.
*/
bool
IsTransactionOrTransactionBlock(void)
{
TransactionState s = CurrentTransactionState;
- if (s->blockState == TBLOCK_DEFAULT)
+ /*
+ * blockState == TBLOCK_DEFAULT usually what indicates that there's no
+ * transaction in progress, but also check the low-level state, in case
+ * we're starting a transaction, or cleaning up a transaction that failed
+ * to start.
+ */
+ if (s->blockState == TBLOCK_DEFAULT && s->state == TRANS_DEFAULT)
return false;
return true;
--
2.47.3
From ca798b7828c655198f1c39030ce901a4437e5979 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Tue, 17 Nov 2026 00:18:18 +0200
Subject: [PATCH 3/3] Add a test for losing a sinval catchup signal
Add an injection point in ReceiveSharedInvalidMessages() after it has
caught up, and a TAP test that stops an idle backend there, makes it
fall behind again so that it's sent a second catchup signal, and checks
that the backend processes the second request after it continues, and
that it's sent further catchup signals after that.
With the extra sleep in the test, this also covers the assertion failure
---
src/backend/storage/ipc/sinval.c | 12 ++
src/test/modules/test_misc/meson.build | 1 +
.../modules/test_misc/t/016_sinval_catchup.pl | 131 ++++++++++++++++++
3 files changed, 144 insertions(+)
create mode 100644 src/test/modules/test_misc/t/016_sinval_catchup.pl
diff --git a/src/backend/storage/ipc/sinval.c b/src/backend/storage/ipc/sinval.c
index 756776c8732..6c5af2951b4 100644
--- a/src/backend/storage/ipc/sinval.c
+++ b/src/backend/storage/ipc/sinval.c
@@ -18,6 +18,7 @@
#include "miscadmin.h"
#include "storage/latch.h"
#include "storage/sinvaladt.h"
+#include "utils/injection_point.h"
#include "utils/inval.h"
@@ -78,6 +79,7 @@ ReceiveSharedInvalidMessages(void (*invalFunction) (SharedInvalidationMessage *m
*/
static volatile int nextmsg = 0;
static volatile int nummsgs = 0;
+ bool read_any = false;
/* Deal with any messages still pending from an outer recursion */
while (nextmsg < nummsgs)
@@ -96,6 +98,8 @@ ReceiveSharedInvalidMessages(void (*invalFunction) (SharedInvalidationMessage *m
/* Try to get some more messages */
getResult = SIGetDataEntries(messages, MAXINVALMSGS);
+ if (getResult != 0)
+ read_any = true;
if (getResult < 0)
{
@@ -124,6 +128,14 @@ ReceiveSharedInvalidMessages(void (*invalFunction) (SharedInvalidationMessage *m
*/
} while (nummsgs == MAXINVALMSGS);
+ /*
+ * Allow tests to pause here, after we have caught up, but before we pass
+ * on any catchup request. Only if we read any messages or a reset, so
+ * that this is not reached on every call.
+ */
+ if (read_any)
+ INJECTION_POINT("sinval-caught-up", NULL);
+
/*
* We are now caught up. If we received a catchup signal, call
* SICleanupQueue(). This is not so much because we need to flush dead
diff --git a/src/test/modules/test_misc/meson.build b/src/test/modules/test_misc/meson.build
index 5d81f5b13be..0e97a511909 100644
--- a/src/test/modules/test_misc/meson.build
+++ b/src/test/modules/test_misc/meson.build
@@ -24,6 +24,7 @@ tests += {
't/013_temp_obj_multisession.pl',
't/014_log_statement_max_length.pl',
't/015_temp_schema_exit_deferrable.pl',
+ 't/016_sinval_catchup.pl',
],
# The injection points are cluster-wide, so disable installcheck
'runningcheck': false,
diff --git a/src/test/modules/test_misc/t/016_sinval_catchup.pl b/src/test/modules/test_misc/t/016_sinval_catchup.pl
new file mode 100644
index 00000000000..5850286de22
--- /dev/null
+++ b/src/test/modules/test_misc/t/016_sinval_catchup.pl
@@ -0,0 +1,131 @@
+
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test that a shared invalidation catchup request is not lost, if it arrives
+# while the backend is still processing a previous catchup request.
+#
+# When a backend falls far behind in the shared invalidation queue,
+# SICleanupQueue() sends it a catchup signal and sets its "signaled" flag.
+# The backend clears the flag when it has caught up, and as long as it is
+# set, no further catchup signals are sent to it. If a new signal arrives
+# after the backend has caught up and cleared the flag, but before it has
+# finished processing the previous request, the new request must not be
+# lost. Otherwise the backend is left with the flag set but no pending
+# request, and won't be sent any more catchup signals while it stays idle.
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+my $node = PostgreSQL::Test::Cluster->new('node');
+$node->init();
+
+# The idle session below should be the backend furthest behind in the
+# invalidation queue, so that it's the one that gets sent catchup requests.
+# Keep autovacuum out of the way.
+$node->append_conf('postgresql.conf', 'autovacuum = off');
+$node->append_conf('postgresql.conf', 'log_min_messages = debug4');
+$node->start;
+
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Start the session that we'll observe. It logs at DEBUG4 so that we can see
+# when it processes catchup requests. Attach the injection point locally,
+# so that only this session waits on it. After the setup, the session sits
+# idle, waiting for a command from the client, which is when it processes
+# catchup requests.
+my $idle = $node->background_psql('postgres');
+my $idle_pid = $idle->query_safe('SELECT pg_backend_pid();');
+$idle->query_safe(
+ q{
+ SELECT injection_points_set_local();
+ SELECT injection_points_attach('sinval-caught-up', 'wait');
+});
+
+my $sent_re = qr/sending sinval catchup signal to PID $idle_pid\b/;
+my $processed_re = qr/\[$idle_pid\] .*DEBUG: ProcessCatchupEvent/;
+
+# Count the catchup requests sent to the idle session so far.
+sub requests_sent
+{
+ my $log = slurp_file($node->logfile);
+ my $count = () = $log =~ /$sent_re/g;
+ return $count;
+}
+
+# Generate invalidation messages, in batches, until the idle session has been
+# sent at least $target catchup requests. The sending backend logs at DEBUG4
+# too, so that we can see the requests in the log.
+sub generate_invals_until_sent
+{
+ my ($target) = @_;
+
+ for (my $i = 0; $i < 1000; $i++)
+ {
+ return 1 if requests_sent() >= $target;
+ $node->safe_psql(
+ 'postgres', q{
+ DO $$
+ BEGIN
+ FOR i IN 1..10 LOOP
+ EXECUTE 'CREATE TABLE sinval_t (a int)';
+ EXECUTE 'DROP TABLE sinval_t';
+ END LOOP;
+ END $$;
+ });
+ }
+ return requests_sent() >= $target;
+}
+
+# Make the idle session fall behind, so that it gets a catchup request. It
+# catches up, and then stops at the injection point, before it has finished
+# processing the request.
+ok(generate_invals_until_sent(1),
+ 'idle session was sent a catchup request');
+$node->wait_for_event('client backend', 'sinval-caught-up');
+
+# While it's stopped there, make it fall behind again, so that it gets a
+# second catchup request. It has already caught up, so it's eligible for a
+# new request.
+ok(generate_invals_until_sent(2),
+ 'idle session was sent a second catchup request while processing the first'
+);
+
+# This sleep makes the test hit the assertion failure
+sleep(10);
+
+# Let it continue. Detach first, so that it doesn't stop again. It should
+# now process the second request too.
+my $log_offset = -s $node->logfile;
+$node->safe_psql(
+ 'postgres', q{
+ SELECT injection_points_detach('sinval-caught-up');
+ SELECT injection_points_wakeup('sinval-caught-up');
+});
+$node->wait_for_log($processed_re, $log_offset);
+pass('idle session processed the second catchup request');
+
+# Once it has caught up again, it should get further catchup requests as
+# usual. If the second request had been lost, its "signaled" flag would
+# still be set, and it would never be sent another one.
+ok(generate_invals_until_sent(3),
+ 'idle session was sent a catchup request again');
+
+# The session is still usable.
+is($idle->query_safe('SELECT 1;'), '1', 'idle session still works');
+$idle->quit;
+
+done_testing();
--
2.47.3