SpongeRobert opened a new issue, #2580:
URL: https://github.com/apache/age/issues/2580
**Versions.** PostgreSQL 18.6, Apache AGE at the head of branch `PG18`,
`e43dc1a12b78fba4acef9835b2b10379b8d243b4` (the 1.8.0 release commit, head on
2026-09-24), `shared_preload_libraries = 'age'`. The extension does not have
to
be created in any database. `is_age_extension_exists()` is the same on
`master`
(`fa109ef1dd`); the check came with `a17bf457fe` ("Make age extension usable
from shared_preload_libraries").
**Symptom.** Backends fail at connect with `FATAL: failed to re-find shared
lock object` (SQLSTATE XX000), in every database and under every role, until
the server restarts. Later the server log may also show
`lock AccessShareLock on object <db>/3079/0 is already held`.
**What happens.**
1. A session fails a statement inside `BEGIN` and waits for `ROLLBACK`
(`TBLOCK_ABORT`: `AbortTransaction()` has run, the resource owner of the
transaction is released).
2. Its caches are reset: the shared invalidation queue overflowed while the
backend was not reading it. The relcache callback clears
`extension_cache_is_valid`, and the catcache entry of `pg_extension` is
gone.
3. `ROLLBACK` is parsed. `post_parse_analyze` calls
`is_age_extension_exists()`,
which calls `get_extension_oid("age", true)`. The catcache miss opens
`pg_extension` and then `pg_class`. Their locks are granted on the fast
path,
then `ResourceOwnerEnlarge()` fails:
`ERROR: ResourceOwnerEnlarge called after release started`.
4. Nothing releases those locks any more. In `TBLOCK_ABORT` the transaction
is
already aborted, so the backend exits with the two relations still in the
fast-path array of its PGPROC.
5. The next backend on that PGPROC takes the same relation. It reuses the
slot
without counting it, and its `LockRelease()` goes to the main lock table,
which has no entry: `FATAL: failed to re-find shared lock object` at
connect.
Its exit does not clear the slot, so every later occupant fails the same
way.
The core does not read the catalog in an aborted transaction. The hook does
so
on every parse, `ROLLBACK` included.
**Evidence.**
- **An assertion of the core.** PostgreSQL 18.6 built from `REL_18_6` with
`--enable-cassert --enable-debug`, AGE at `e43dc1a`, no patch. The first
reproduction run aborts backend A on its `ROLLBACK`:
```
TRAP: failed Assert("IsTransactionState()"), File: "relcache.c", Line: 2073
postgres: ... ROLLBACK(ExceptionalCondition+0xa8)
postgres: ... ROLLBACK(AssertCouldGetRelation+0x2e)
...
postgres: ... ROLLBACK(SearchCatCache+0x45)
postgres: ... ROLLBACK(SearchSysCache+0x95)
postgres: ... ROLLBACK(GetSysCacheOid+0x3e)
postgres: ... ROLLBACK(get_extension_oid+0x40)
age.so(+0x11d3f)
age.so(+0x2900f)
postgres: ... ROLLBACK(parse_analyze_fixedparams+0xe5)
LOG: client backend (PID 109) was terminated by signal 6: Aborted
DETAIL: Failed process was running: ROLLBACK;
```
`AssertCouldGetRelation()` is documented as "In code that reads catalogs in
the event of a cache miss, call this before checking the cache". The same
build with the fix below gives no TRAP, no error on `ROLLBACK` and no
XX000.
- **The stack on a release build at the first error.** Taken with gdb on the
backend of session A, with pgdg debug symbols; the breakpoint is `errstart`
at level `ERROR`:
```
#0 errstart (elevel=21, domain=0x0) at elog.c:354
#1 errstart_cold (elevel=21, domain=0x0) at elog.c:329
#2 ResourceOwnerEnlarge (owner=<optimized out>) at resowner.c:469
#3 RelationIncrementReferenceCount (rel=0x795a669e11b8) at relcache.c:2191
#4 RelationIdGetRelation (relationId=1259) at relcache.c:2121
#5 relation_open (relationId=1259, lockmode=1) at relation.c:58
#6 table_open (relationId=1259, lockmode=1) at table.c:44
#7 ScanPgRelation (targetRelId=3079, indexOK=true,
force_non_historic=false) at relcache.c:371
#8 RelationBuildDesc (targetRelId=<optimized out>, insertIt=true) at
relcache.c:1115
#9 RelationIdGetRelation (relationId=3079) at relcache.c:2144
#10 relation_open (relationId=3079, lockmode=1) at relation.c:58
#11 table_open (relationId=<optimized out>, lockmode=1) at table.c:44
#12 SearchCatCacheMiss (cache=0x5c875436e300, nkeys=1, ...) at
catcache.c:1554
#13 SearchCatCacheInternal (...) at catcache.c:1499
#14 SearchCatCache (...) at catcache.c:1357
#15 SearchSysCache (key1=1, cacheId=<optimized out>) at syscache.c:217
#16 GetSysCacheOid (cacheId=27, oidcol=1, ...) at syscache.c:459
#17 get_extension_oid (extname="age", missing_ok=true) at extension.c:192
#18 is_age_extension_exists () at src/backend/catalog/ag_catalog.c:88
#19 is_age_extension_exists () at src/backend/catalog/ag_catalog.c:74
#20 post_parse_analyze (pstate=..., query=..., jstate=<optimized out>) at
src/backend/parser/cypher_analyze.c:90
#21 parse_analyze_fixedparams (..., sourceText="ROLLBACK;", ...) at
analyze.c:128
#22 pg_analyze_and_rewrite_fixedparams (...) at postgres.c:683
#23 exec_simple_query (query_string="ROLLBACK;") at postgres.c:1190
```
- **The reproduction** (`repro.sh` below: psql, pgbench and kill, on a fresh
server). Ten fresh servers per row:
| Build | `rollback` | `quit` (control: the session leaves without
ROLLBACK) |
|---|---|---|
| AGE `e43dc1a` as is | 10 of 10 reproduced | 0 of 10 |
| the same with the fix below | 0 of 10 | not run |
| AGE `master` (`fa109ef1dd`) as is | 10 of 10 reproduced | not run |
| `master` with the fix | 0 of 10 | not run |
Without the overflow of the invalidation queue (step 2) it is 0 of 10: the
cache of `is_age_extension_exists()` is still valid, so the catalog is not
read.
**Suggested fix** (`fix.diff` below, against `e43dc1a`; it applies to
`master` with an offset of 2 lines, and `make installcheck` on `master` with
it
passes all 42 tests). A pull request against `master` follows. At the top of
`is_age_extension_exists()`:
```c
if (!IsTransactionState())
return false;
```
Nothing is cached, so the next live transaction asks the catalog again. In an
aborted transaction only the statements that leave it reach the parser, and
AGE
has nothing to analyse in them, so "no extension" is the right answer for
that
parse.
The other callers of `is_age_extension_exists()` are `is_age_drop()` (a
`DROP` statement), `object_access()` (DDL) and `set_rel_pathlist()`
(planning). By our reading of `exec_simple_query()`, an aborted transaction
rejects those statements before parse analysis, so only `post_parse_analyze`
reaches the catalog there. That is from reading the code: we reproduced and
traced only the `post_parse_analyze` path. The fix covers every caller
anyway.
Related: #2520, another hook of AGE that misbehaves when `age` is in
`shared_preload_libraries`.
<details><summary><code>repro.sh</code></summary>
```bash
#!/usr/bin/env bash
# Reproduction: the post_parse_analyze hook of Apache AGE reads pg_extension
# while parsing the ROLLBACK of a failed transaction; the locks it takes leak
# into the PGPROC, and the next backend on that PGPROC fails at connect with
# "FATAL: failed to re-find shared lock object".
#
# Needs: a FRESH server with shared_preload_libraries = 'age' (the extension
# does not have to be created), a superuser, psql and pgbench, and a shell on
# the server host as the OS user of the server (step 2 stops a backend with
# kill). Connection parameters come from the usual PG* environment variables.
# A poisoned PGPROC lives until the server restarts: use a throwaway server.
#
# Exit code: 1 when XX000 was seen (reproduced), 0 when every connection of
# step 4 succeeded, 3 when the run could not be carried out.
set -uo pipefail
export PGUSER="${PGUSER:-postgres}" PGDATABASE="${PGDATABASE:-postgres}"
END="${1:-rollback}" # rollback | quit (the control: the session leaves
without ROLLBACK)
fifo=$(mktemp -u); out=$(mktemp); mkfifo "$fifo"
trap 'rm -f "$fifo" "$out" /tmp/age-repro-one.sql' EXIT
# 1. Session A fails a statement inside a transaction and stays in the
# failed block; its input is a fifo, so it stays open between the steps.
psql -XAt <"$fifo" >"$out" 2>&1 &
exec 3>"$fifo"
echo "SELECT pg_backend_pid();" >&3
for _ in $(seq 50); do pid=$(grep -m1 -E '^[0-9]+$' "$out") && break; sleep
0.1; done
[ -n "${pid:-}" ] || { echo "session A did not report its pid"; exit 3; }
echo "BEGIN; SELECT 1/0;" >&3
sleep 0.3
# 2. Its backend is stopped while another session overflows the shared
# invalidation queue; on SIGCONT backend A resets its caches, the cached
# "is age installed" flag and the catcache entry of pg_extension among
them.
kill -STOP "$pid"
psql -Xq -c "DO \$\$ BEGIN
FOR i IN 1..1000 LOOP EXECUTE format('CREATE TABLE age_repro_%s (a
int)', i); END LOOP;
FOR i IN 1..1000 LOOP EXECUTE format('DROP TABLE age_repro_%s', i); END
LOOP;
END \$\$" >/dev/null || { kill -CONT "$pid"; echo "the overflow did not
run"; exit 3; }
kill -CONT "$pid"
sleep 0.3
# 3. ROLLBACK is parsed: the hook reads pg_extension outside a live
transaction
# and fails with "ResourceOwnerEnlarge called after release started".
[ "$END" = rollback ] && echo "ROLLBACK;" >&3
echo '\q' >&3
exec 3>&-
wait
grep -E 'ResourceOwnerEnlarge|ERROR' "$out" || true
# 4. Every free PGPROC is taken at once (the free list is FIFO, so the one
# session A gave back is taken last).
max=$(psql -XAtc "SHOW max_connections")
used=$(psql -XAtc "SELECT count(*) FROM pg_stat_activity WHERE backend_type
= 'client backend'")
n=$((max - used))
echo 'SELECT 1;' >/tmp/age-repro-one.sql
res=$(pgbench -n -c "$n" -j 8 -t 1 -f /tmp/age-repro-one.sql 2>&1)
if grep -qE 'failed to re-find shared (lock|proclock) object' <<<"$res"; then
grep -m1 -E 'failed to re-find' <<<"$res"
echo "REPRODUCED (end=$END, connections=$n)"; exit 1
fi
if grep -q "actually processed: $n/$n\$" <<<"$res"; then
echo "not reproduced (end=$END, connections=$n)"; exit 0
fi
echo "the saturation did not run as intended:"; tail -3 <<<"$res"; exit 3
```
</details>
<details><summary><code>fix.diff</code> (against
<code>e43dc1a</code>)</summary>
```diff
--- a/src/backend/catalog/ag_catalog.c
+++ b/src/backend/catalog/ag_catalog.c
@@ -75,6 +75,15 @@
{
static bool callback_registered = false;
+ /*
+ * Outside a live transaction (the ROLLBACK of a failed one) the catalog
+ * must not be read: the resource owner is already released, and the
locks
+ * a catalog scan takes here would leak into the PGPROC. Nothing is
cached,
+ * so the next live transaction asks the catalog again.
+ */
+ if (!IsTransactionState())
+ return false;
+
if (extension_cache_is_valid)
return age_extension_exists;
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
