Matthias van de Meent <[email protected]> wrote:
> On Mon, 20 Jul 2026 at 19:48, Antonin Houska <[email protected]> wrote:
> >
> > Michael Paquier <[email protected]> wrote:
> >
> > > On Fri, Jul 17, 2026 at 12:00:01PM +0300, Alexander Lakhin wrote:
> > > > 4) make_new_heap():
> > > > --- a/src/backend/commands/repack.c
> > > > +++ b/src/backend/commands/repack.c
> > > > @@ -1255,3 +1255,5 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace,
> > > > Oid NewAccessMethod,
> > > > */
> > > > +oom_prob = 0.01;
> > > > CommandCounterIncrement();
> > > > +oom_prob = 0;
> > > >
> > > > triggers:
> > > > TRAP: failed Assert("ItemIdIsNormal(lp)"), File: "heapam.c", Line:
> > > > 2813, PID: 943626
> > > > also:
> > > > ERROR: attempted to delete invisible tuple
> > >
> > > I'll let Alvaro comment on this one. It deserves an open item.
> >
> > I was also curious because I worked on repack.c recently. After applying (a
> > bit rebased) random-oom-errors.patch from [1] and after turning some "out of
> > memory" errors to PANIC, I could reproduce the error even on PG 18. The
> > stack
> > trace is below. So it does not seem to be PG 19 specific.
>
> A PANIC with AllocSetAlloc on the stacktrace isn't an indication of a
> bug when you add both an OOM probability on palloc, and set the
> OOM-error-thrower to PANIC instead of ERROR.
>
> This thread contains various examples of backend-local state remaining
> in various levels of incomplete modifications/corruptions even after
> the ERROR handling of an OOM has been completed. PANICs cause the
> backend to be terminated, and so are much less likely to cause
> problems down the line.
>
> Note, too, that the problem that Alexander reported is a failing
> Assert(), and an ERROR reporting "attempted to delete invisible
> tuple", rather than actual "out of memory" ERRORs.
I supposed that these problems happen because relcache remains in wrong state
due to the "out of memory" ERROR. So the point of using PANIC was to find out
when exactly the cache gets broken. However, I missed the fact that
Alexander's patch triggers the ERROR at several different places, so the stack
trace I posted might be unrelated.
Nevertheless, on a closer look I concluded that the problem is not related to
OOM. Rather, *any* ERROR that happens during cache invalidation can cause
this. Attached here is cache_inval_failure.diff that reproduces the problem
reliably:
postgres=# CREATE TABLE t1(i int PRIMARY KEY); CLUSTER t1 USING t1_pkey; DROP
TABLE t1;
CREATE TABLE
ERROR: failure during cache invalidation
ERROR: attempted to delete invisible tuple
(The Assert(ItemIdIsNormal(lp)) IMO fires only if VACUUM manages to reset the
item pointer before heap_delete() gets called. With autovacuum=off, I get only
the "failure during cache invalidation" error.)
So I think we need to make sure that ERROR during cache invalidation becomes
FATAL.
--
Antonin Houska
Web: https://www.cybertec-postgresql.com
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index 94a5aee90b5..7d9812804ce 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -784,7 +784,9 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, Oid NewAccessMethod,
* Advance command counter so that the newly-created relation's catalog
* tuples will be visible to table_open.
*/
+ invalidation_failure = true;
CommandCounterIncrement();
+ invalidation_failure = false;
/*
* If necessary, create a TOAST table for the new relation.
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index 30048e153e5..ca8a16c50f6 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -180,6 +180,8 @@ typedef struct InvalMessageArray
static InvalMessageArray InvalMessageArrays[2];
+bool invalidation_failure = false;
+
/* Control information for one logical group of messages */
typedef struct InvalidationMsgsGroup
{
@@ -1419,6 +1421,9 @@ CommandEndInvalidationMessages(void)
ProcessInvalidationMessages(&transInvalInfo->ii.CurrentCmdInvalidMsgs,
LocalExecuteInvalidationMessage);
+ if (invalidation_failure)
+ elog(ERROR, "failure during cache invalidation");
+
/* WAL Log per-command invalidation messages for wal_level=logical */
if (XLogLogicalInfoActive())
LogLogicalInvalidations();
diff --git a/src/include/utils/inval.h b/src/include/utils/inval.h
index b5fbd754c2c..1512e47d38d 100644
--- a/src/include/utils/inval.h
+++ b/src/include/utils/inval.h
@@ -20,6 +20,8 @@
extern PGDLLIMPORT int debug_discard_caches;
+extern bool invalidation_failure;
+
typedef void (*SyscacheCallbackFunction) (Datum arg, int cacheid, uint32 hashvalue);
typedef void (*RelcacheCallbackFunction) (Datum arg, Oid relid);
typedef void (*RelSyncCallbackFunction) (Datum arg, Oid relid);
