Hi Greg,
> Attached is v4 which I hope fixes the various cf-bot build failures.
It clears the ones I hit. I measured v4 on master b28a2b2bfa5, against v2 for
the FILEOPS cost, and checked the per-prefix test claim. Two findings, plus an
adversarial pass that came up clean.
1. ROLLBACK and crash recovery now pay two fsyncs per FILEOPS undo record
fileops_sync_inverse() fsyncs the file and its parent after every applied
inverse, and since FILEOPS can no longer be deferred that path runs inline at
ROLLBACK and in the crash-recovery undo phase. N inverses on one file is 2N
fsyncs. For N chmods in one transaction (ROLLBACK timed, median of 3), the
rollback goes from 2.5 ms on v2 to 660 ms on v4 at N=1000, and from 77 ms to
22.8 s at N=32000. Crash-recovery undo of N in-flight records, from the
server's own log: N=8000 is about 7 ms on v2 and about 5.6 s on v4. With
fsync=off, v4 at N=1000 drops from 638 ms to 1.4 ms, so the cost is entirely
the fsyncs, not asserts; the forward direction (doing the N chmods) is
unchanged from v2.
Durability of the inverse before the CLR is the point, but it can be batched:
fsync each distinct path (and its parent) once per chain, before the record
that marks the chain done, instead of once per record. As it stands a
transaction that touched one file 10k times takes about 7 s to roll back and
adds about 7 s of undo before the cluster opens after a crash.
2. "every prefix builds and passes its own tests" does not quite hold
All of 0001-0008 build without warnings and pass make check. Their own module
tests do not all run at their prefix:
- 0002: test_fileops/001 sets max_logical_revert_workers, a GUC added in 0005,
so the server will not start -- the same shape as the split issue Shihao
reported. test_fileops is not listed in src/test/modules until 0005, so the
build farm does not run it there.
- 0004: six recovery tests need the flux AM (0019) or test_idxundo (0006).
- 0005: eight test_idxundo TAP files ship here but the module arrives in 0006.
And test_fileops/016 surfaces a real issue at this prefix: with an injected
fsync failure on an inverse during abort, the backend raises ERROR inside
abort, re-enters abort five times, and ends in PANIC: ERRORDATA_STACK_SIZE
exceeded; the PANIC elevel that makes it one clean failure arrives in 0007.
On the full series, 016 passes.
- 0008: recovery 094 and 121 need the flux AM.
The full list with the exact files is in the attached notes.
3. Adversarial pass: nothing to report
On an ASan+UBSan build I ran crash injection across plain ROLLBACK, savepoints,
postmaster and backend kills, a kill during ROLLBACK, a crash inside the
recovery undo phase, and 2PC. Each time the file tree came back exactly to the
committed baseline, with no sanitizer report and no assert (a COMMIT control
diverges, so the check is live).
Repros and the full measurements are attached.
Regards,
Manu
# UNDO/FILEOPS v3/v4 review: evidence (2026-10-13)
Target: **v4** (Greg, 2026-10-13 00:27, "Re: UNDO with FILEOPS, and the FLUX
table AM"), which supersedes v3 (2026-10-12) one day later; 0002, 0003 and
0006-0008 are byte-identical between v3 and v4 (see PATCH_MAP.md). Applied with
`git am` on master **b28a2b2bfa5** (2026-10-13), all 20 patches clean. Trees in
~/.cache/ctr-v3: normal (-O2), cassert (-O2 + asserts, same flags as the v2
install ~/undo-v2-inst used for the v2 numbers), asan (ASan+UBSan, gcc 16).
Machine: Ryzen AI 9 HX 370, NVMe + btrfs. Lane A CPUs, integrity sentinel clean.
## Finding 1 (measured, new in v3): ROLLBACK and crash recovery pay 2 fsyncs
per FILEOPS UNDO record
v3 added `fileops_sync_inverse()` (fileops_undo.c ~697): after **every**
applied inverse it fsyncs the file **and** its parent directory (elevel PANIC).
The same apply path runs inline at ROLLBACK (FILEOPS can no longer be deferred,
`needs_inline_abort`, xactundo.c) and in the crash-recovery undo phase before
the cluster opens. N inverses on the same file -> 2N fsyncs of the same two
inodes.
A) Client-visible ROLLBACK latency, N chmods in one txn, only ROLLBACK timed,
median of 3, every trial validated (file mode back to baseline). Same
workload as our v2 report (logs/bench/rollback_raw.tsv, summary.txt):
threshold build N=200 N=1000 N=8000 N=32000 (ms)
65536 v2 0.64 2.47 19.6 77.0
65536 v4c 132 660 5822 22851
1e9 (inline) v4c 147 752 5423 21748
1 (defer) v4c 154 738 5870 23007
~0.7 ms per record (= this NVMe's fsync cost) vs ~2.4 us in v2: ~300x.
undo_instant_abort_threshold no longer changes anything for FILEOPS (by
design in v3: FILEOPS UNDO is always inline).
B) Discriminator (logs/fsync_discriminator.txt), N=1000, median of 3:
build fsync=on fsync=off
v2 1.9 ms 1.8 ms
v4n 638 ms 1.4 ms <- no asserts: the whole cost is fsync
v4c 1006 ms 1.5 ms
C) Crash with an in-flight txn of N FILEOPS records, immediate stop, restart;
undo phase timed from the server's own log; undo phase completes before
"ready to accept connections" in every trial; records applied == N
(logs/bench/recovery_raw.tsv, recovery_v4_raw.tsv):
N v2 v4n v4c
2000 2 ms 1318 ms 1450 ms
8000 7-8 ms 5570 ms 5640 ms
32000 28-39 ms (~22 s extrapolated, not run)
So the v2 point "linear before open" still holds, with a ~700x larger slope.
Forward cost (doing the N chmods) is ~0.9 ms/op in BOTH v2 and v4
(logs/forward_cost.txt): the regression is entirely on the undo side.
Redo of the same records is also fsync-bound: a crash during ROLLBACK of
6000 records spent 7.7 s in redo (ASan build, logs/adv/S5b.server.log).
Why it matters / what to ask: durability of the inverse before the
abort/CLR is the point of the v3 fix, but it can be batched: fsync each
distinct path (and parent) once per chain, before the record that marks the
chain done, instead of once per record. As is, a txn that touched one file
10k times takes ~7 s to ROLLBACK and adds ~7 s of undo before open after a
crash. Repro: scripts/repro_rollback_fsync.sql (+ fsync_discriminator.sh).
## Finding 2 (measured): "every prefix builds and passes its own tests" does
not hold
Clean checkout + clean configure (cassert, TAP, injection points) per prefix,
core `make check`, then the TAP/regress files the patch adds or modifies
(scripts/prefix_check.sh, logs/prefix/). Every prefix 0001-0008 **builds with 0
warnings and passes core make check (239/239)**. Own tests:
- 0001: PASS (pg_waldump/001).
- 0002: **FAIL** test_fileops/t/001_fileops_apply_failure.pl: sets
`max_logical_revert_workers = 0`, a GUC added in 0005 -> "unrecognized
configuration parameter", pg_ctl start fails. Same shape as Shihao's #1
(059/061 setting a 0005 GUC). Not seen by the build farm/meson because
test_fileops is not listed in src/test/modules/{Makefile,meson.build} until
0005 (logs/meson_audit.txt). Recovery 060-063, 072: PASS.
- 0003: no own tests; build + core PASS.
- 0004: **FAIL** 6 tests die with `access method "flux" does not exist`
(FLUX = 0019): recovery 073, 074, 075, 083, 089 and test_checksums/025 (all
registered in meson at this prefix). recovery/114 needs
shared_preload_libraries=test_idxundo (module added in 0006): pg_ctl start
fails and prove bails out (115 [also FLUX], 117, 152 not reached).
085/096 fail on pg_walinspect/amcheck not installed under the autoconf
harness (recovery Makefile EXTRA_INSTALL not extended; meson installs all
contrib) - minor, autoconf-only.
- 0005: **8 orphan TAP files**: test_idxundo/t/002-009 ship here but the module
(Makefile, meson.build, .c) arrives in 0006 - nothing can run them.
test_fileops/005 dies: reloption `index_undo` (0006). **test_fileops/016
FAILS 3/5 and shows real misbehaviour at this prefix**: with an injected
fsync failure on an inverse during abort, the backend raises ERROR inside
abort, re-enters abort 5 times (5 abort records) and ends in
`PANIC: ERRORDATA_STACK_SIZE exceeded` instead of one clean PANIC. The
elevel=PANIC for that path is only introduced in 0007 (the hash patch).
On the full series 005 and 016 PASS (verified).
- 0006, 0007: build + core PASS (own tests not run in this pass).
- 0008: **FAIL** recovery 094 and 121 (registered in meson here) die with
`access method "flux" does not exist`.
Repro: scripts/repro_prefix_tests.sh (0002 and 0004 cases).
## Adversarial pass (ASan+UBSan, crash injection): nothing found
scripts/adv_fileops.sh on inst-asan. Victim txn = 13 mixed FILEOPS ops
(chmod, write, truncate, create, mkdir, create-in-new-dir, link, symlink,
setxattr, removexattr, rename) + NPAD chmods to widen windows. Invariant: test
tree (type, mode, size, md5, link target, user xattrs) == committed baseline;
server log has no PANIC/TRAP/assert; no ASan/UBSan report.
S0 negative control (same ops COMMITTED) DIVERGED (harness valid)
S1 plain ROLLBACK OK
S2 SAVEPOINT / ROLLBACK TO + more ops, ROLLBACK OK
S3 in-flight, postmaster immediate stop OK (3011 records undone
before open)
S4 in-flight, kill -9 backend (crash-restart) OK (3011 records)
S5 kill -9 backend DURING its ROLLBACK OK ("Failed process was
running: ROLLBACK")
S5b same, snapshot at FIRST accepted connection already reverted at open
(redo finished it)
S6 crash DURING the recovery undo phase, restart OK (1.8 s into phase;
2nd recovery re-applies
the whole chain, 6011
records, idempotent)
S7 PREPARE TRANSACTION, crash, ROLLBACK PREPARED OK (prepared xact
skipped by undo, kept)
No sanitizer report in any scenario. (The 6 -Wnonnull warnings of the ASan
build are gcc-16 false positives in upstream code: path.c, indexcmds.c,
tablecmds.c, md.c.) Also confirmed by construction: v3's alignment bug (undo
record headers at non-MAXALIGN'd offsets) is what v4 fixes; we tested v4.
Harness notes (cost us time, not findings): FILEOPS rename is deferred to
commit, so writing to the new name inside the txn errors out (the first S1/S2
"OK" were vacuous until S0 caught it); `pg_ctl start` waits by default (-W
needed to crash inside the undo phase).
## Not done in this pass
- 0006/0007 own TAP suites; 0005 recovery 090/126/134 and test_fileops
002-009 at that prefix.
- Deeper adversarial (ENOSPC/EIO during inline apply -> ATM fallback path at
xactundo.c ~1183, CREATE DATABASE crash windows, standby).
- v2-vs-v4 ABBA second round (effect size 300x makes it moot).
-- Minimal repro: v3/v4 ROLLBACK of N FILEOPS operations costs ~N x (fsync of
-- the file + fsync of its parent directory), applied inline before ROLLBACK
-- returns (fileops_sync_inverse() after every inverse). Same with crash-loser
-- undo before the cluster opens. Run with psql against a v4 build:
-- CREATE EXTENSION test_fileops; then \i this file, once with fsync=on and
-- once after ALTER SYSTEM SET fsync=off; SELECT pg_reload_conf();
-- Measured (NVMe, btrfs), N=1000: v2 1.9 ms | v4 (no cassert) 638 ms with
-- fsync=on, 1.4 ms with fsync=off.
SELECT test_fileops_create_tempfile('fsync_repro.dat') AS f \gset
BEGIN;
SELECT count(test_fileops_chmod(:'f', 384 + (g % 2) * 36)) FROM
generate_series(1, 1000) g;
\timing on
ROLLBACK;
\timing off
#!/usr/bin/env bash
# Minimal repro: patches' own tests depend on later patches.
# repro_prefix_tests.sh <postgres-src-with-series-as-commits> <tag-0002>
<tag-0004>
# 1) at 0002: test_fileops/t/001 sets max_logical_revert_workers (added in 0005)
# -> FATAL: unrecognized configuration parameter; pg_ctl start fails.
# 2) at 0004: recovery/t/073 (and 074,075,083,089, test_checksums/025) do
# CREATE TABLE ... USING flux (added in 0019)
# -> ERROR: access method "flux" does not exist.
# recovery/t/114 needs shared_preload_libraries=test_idxundo (added in 0006).
set -eu
SRC=$1; T2=$2; T4=$3; W=$(mktemp -d)
run() { # $1 tag, $2 test dir, $3 tests
git -C "$SRC" worktree add -q --detach "$W/$1" "$1"
(cd "$W/$1" && ./configure -q --enable-cassert --enable-tap-tests
--enable-injection-points \
--without-icu >/dev/null && make -s -j8 >/dev/null && make -s -C contrib
-j8 >/dev/null \
&& make -C "$2" check PROVE_TESTS="$3" 2>&1 | tail -5)
grep -hE 'FATAL|ERROR' "$W/$1/$2"/tmp_check/log/* | head -3
}
run "$T2" src/test/modules/test_fileops t/001_fileops_apply_failure.pl
run "$T4" src/test/recovery t/073_perbackend_2pc_rollback.pl
