Hi Greg,

The four fixes hold: your tests pass and each fails when I revert its
fix.  Two cases outside the window you document still end with files
that differ from what the committed transactions imply; the attached
oracle runs 186 scenarios.

On Wed, Nov 11, 2026 at 1:09 PM Greg Burd <[email protected]> wrote:
> The abort record carries the transaction's FILEOPS inverses, and abort
> redo applies them, so a standby, and the primary after a crash, end up
> with the files the primary had after ROLLBACK

Not for a crash loser: the primary's startup undo phase removes its
files, but a crash writes no abort record, so a streaming standby keeps
them until it is promoted.  I see it for create, mkdir, link, rename and
write, and on v9 for the two I tried there.  After a link, if another
transaction then creates that name, the standby's source file ends up
with that transaction's bytes.

FileOpsCreate() opens without O_EXCL and writes the create UNDO even if
the file existed, so a rollback unlinks a committed file, on the
primary, the standby and after a crash.  No in-core caller does that;
the first diff makes it an ERROR, as FileOpsMkdir() already does.

The hold-off in InsertXactUndoData() and ROLLBACK PREPARED keeping
DELAY_CHKPT_START have no test that fails without them.  The second
diff adds one for each (the hold-off one drives gdb, so take it as a
repro rather than a test for the tree), the crash-loser repro, and a
copy of 024 with three failures.

> Perhaps I should separate them into a 4th patch to make that explicit.

In 0001 the subtransaction and 2PC functions are about 140 of the 1,626
lines gcov counts; at most 58 lines run only under the tests using them.

Manu
diff --git a/src/backend/storage/file/README b/src/backend/storage/file/README
index 26c9a5b5120..ed29ae8bbee 100644
--- a/src/backend/storage/file/README
+++ b/src/backend/storage/file/README
@@ -66,7 +66,8 @@ These run when the API is called, so the file or directory 
can be used
 within the same transaction, and each registers an abort-time undo action
 that reverses it if the transaction rolls back:
 
-  FileOpsCreate    -- create() now; UNDO/pending-delete on abort
+  FileOpsCreate    -- create() now, an ERROR if the file exists; 
UNDO/pending-delete
+                      on abort
   FileOpsMkdir     -- mkdir() now; UNDO rmdir on abort
   FileOpsMkdirTree -- mkdir() now; UNDO removes the whole tree on abort
   FileOpsChmod     -- chmod() now; UNDO restore old mode on abort
diff --git a/src/backend/storage/file/fileops.c 
b/src/backend/storage/file/fileops.c
index 386eb618910..8ad7ab281ff 100644
--- a/src/backend/storage/file/fileops.c
+++ b/src/backend/storage/file/fileops.c
@@ -431,6 +431,11 @@ FileOpsCancelPendingDelete(const char *path, bool 
at_commit)
  * and logs the creation to WAL. If register_delete is true, the file will
  * be deleted if the transaction aborts.
  *
+ * A path that already exists is an ERROR (ERRCODE_DUPLICATE_FILE), as for
+ * FileOpsMkdir(), FileOpsLink() and FileOpsRename().  The UNDO record
+ * unlinks the path, so the path must name a file this call created: opening
+ * an existing file here would let a rollback remove a committed one.
+ *
  * Platform handling:
  *   - Linux/FreeBSD: O_DIRECT passed directly to open()
  *   - macOS: F_NOCACHE fcntl applied after open()
@@ -446,7 +451,7 @@ FileOpsCreate(const char *path, int flags, mode_t mode, 
bool register_delete)
 
        Assert(!IsInParallelMode());
 
-       fd = OpenTransientFilePerm(path, flags | O_CREAT, mode);
+       fd = OpenTransientFilePerm(path, flags | O_CREAT | O_EXCL, mode);
        if (fd < 0)
                ereport(ERROR,
                                (errcode_for_file_access(),
diff --git a/src/test/modules/test_fileops/meson.build 
b/src/test/modules/test_fileops/meson.build
index e14676912a4..e9c51446bed 100644
--- a/src/test/modules/test_fileops/meson.build
+++ b/src/test/modules/test_fileops/meson.build
@@ -44,6 +44,7 @@ tests += {
       't/018_fileops_rename.pl',
       't/023_fileops_inline_abort_ctx.pl',
       't/024_fileops_recovery_apply_failure.pl',
+      't/025_fileops_create_existing.pl',
     ],
   },
 }
diff --git a/src/test/modules/test_fileops/t/025_fileops_create_existing.pl 
b/src/test/modules/test_fileops/t/025_fileops_create_existing.pl
new file mode 100644
index 00000000000..a0db2dc29f0
--- /dev/null
+++ b/src/test/modules/test_fileops/t/025_fileops_create_existing.pl
@@ -0,0 +1,56 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# FileOpsCreate() on a path that already exists, then ROLLBACK.  The UNDO
+# record of a create unlinks the path, so if the call opened an existing file
+# a rollback would remove a committed one.  The committed file must survive
+# on the primary, on a standby replaying the abort record, and on the primary
+# after crash recovery replays it.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+$primary->start;
+$primary->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+$primary->backup('backup');
+my $standby = PostgreSQL::Test::Cluster->new('standby');
+$standby->init_from_backup($primary, 'backup', has_streaming => 1);
+$standby->start;
+
+$primary->safe_psql('postgres', q{
+BEGIN;
+SELECT test_fileops_create('existing.dat', 384);
+SELECT test_fileops_write('existing.dat', 0, 'committed');
+COMMIT;
+});
+
+my ($ret, $out, $err) = $primary->psql('postgres', q{
+BEGIN;
+SELECT test_fileops_create('existing.dat', 384);
+ROLLBACK;
+});
+like($err, qr/could not create file "existing.dat": File exists/,
+       'creating an existing path is an ERROR');
+
+$primary->wait_for_catchup($standby);
+for my $stage ('online', 'after crash')
+{
+       if ($stage eq 'after crash')
+       {
+               $primary->stop('immediate');
+               $primary->start;
+       }
+       my $f = $primary->data_dir . '/existing.dat';
+       is(-e $f ? slurp_file($f) : '(missing)',
+               'committed', "$stage: primary keeps the committed file");
+}
+my $sf = $standby->data_dir . '/existing.dat';
+is(-e $sf ? slurp_file($sf) : '(missing)',
+       'committed', 'standby keeps the committed file');
+
+$standby->stop;
+$primary->stop;
+done_testing();
diff --git a/src/test/modules/test_fileops/meson.build 
b/src/test/modules/test_fileops/meson.build
index e14676912a4..f6e4d7fc69c 100644
--- a/src/test/modules/test_fileops/meson.build
+++ b/src/test/modules/test_fileops/meson.build
@@ -44,6 +44,10 @@ tests += {
       't/018_fileops_rename.pl',
       't/023_fileops_inline_abort_ctx.pl',
       't/024_fileops_recovery_apply_failure.pl',
+      't/026_fileops_abort_checkpoint_order.pl',
+      't/027_fileops_crash_loser_standby.pl',
+      't/028_fileops_recovery_apply_failure_count.pl',
+      't/029_fileops_insert_holdoff_gdb.pl',
     ],
   },
 }
diff --git 
a/src/test/modules/test_fileops/t/026_fileops_abort_checkpoint_order.pl 
b/src/test/modules/test_fileops/t/026_fileops_abort_checkpoint_order.pl
new file mode 100644
index 00000000000..0e68ae43398
--- /dev/null
+++ b/src/test/modules/test_fileops/t/026_fileops_abort_checkpoint_order.pl
@@ -0,0 +1,75 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# A checkpoint must not move the redo pointer past an abort record whose
+# FILEOPS inverses are not yet durable.  Pause the aborting backend after the
+# abort record is flushed and before it applies the inverses, request a
+# CHECKPOINT, then crash.  Recovery must replay the abort record and remove
+# the file, for ROLLBACK and for ROLLBACK PREPARED.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+plan skip_all => 'injection points not supported by this build'
+  unless $ENV{enable_injection_points} eq 'yes';
+
+my $node = PostgreSQL::Test::Cluster->new('n');
+$node->init(allows_streaming => 1);
+$node->append_conf('postgresql.conf',
+       "max_prepared_transactions = 5\ncheckpoint_timeout = 1h\n");
+$node->start;
+$node->safe_psql('postgres',
+       'CREATE EXTENSION test_fileops; CREATE EXTENSION injection_points');
+
+for my $case ('rollback', 'rollback prepared')
+{
+       (my $f = "ck-$case.dat") =~ s/ /-/g;
+       $node->safe_psql('postgres',
+               "SELECT injection_points_attach('fileops-abort-after-flush', 
'wait')");
+       my $before = $node->safe_psql('postgres',
+               'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+
+       my $a = $node->background_psql('postgres');
+       if ($case eq 'rollback')
+       {
+               $a->query_until(qr/go/, qq{\\echo go
+BEGIN; SELECT test_fileops_create('$f', 384); ROLLBACK;
+});
+       }
+       else
+       {
+               $node->safe_psql('postgres',
+                       "BEGIN; SELECT test_fileops_create('$f', 384); PREPARE 
TRANSACTION 'p';");
+               $a->query_until(qr/go/, qq{\\echo go
+ROLLBACK PREPARED 'p';
+});
+       }
+       $node->wait_for_event('client backend', 'fileops-abort-after-flush');
+       ok(-e $node->data_dir . "/$f", "$case: file still there at the pause");
+
+       my $c = $node->background_psql('postgres');
+       $c->query_until(qr/go/, "\\echo go\nCHECKPOINT;\n");
+
+       # Give the checkpoint time to finish if nothing holds it back.
+       my $after = $before;
+       for (1 .. 50)
+       {
+               $after = $node->safe_psql('postgres',
+                       'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+               last if $after ne $before;
+               usleep(100_000);
+       }
+       note "$case: checkpoint completed during the pause: "
+         . ($after ne $before ? 'yes' : 'no');
+
+       $node->stop('immediate');
+       eval { $a->quit; };
+       eval { $c->quit; };
+       $node->start;
+       ok(!-e $node->data_dir . "/$f", "$case: crash recovery removed the 
file");
+}
+
+$node->stop;
+done_testing();
diff --git a/src/test/modules/test_fileops/t/027_fileops_crash_loser_standby.pl 
b/src/test/modules/test_fileops/t/027_fileops_crash_loser_standby.pl
new file mode 100644
index 00000000000..04d77a3f360
--- /dev/null
+++ b/src/test/modules/test_fileops/t/027_fileops_crash_loser_standby.pl
@@ -0,0 +1,63 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# A transaction is in flight with a FILEOPS create when the primary crashes.
+# The primary's startup undo phase reverses it.  A crashed transaction leaves
+# no abort record, so the question is whether the standby, replaying the
+# primary's WAL, ends with the same files.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+$primary->append_conf('postgresql.conf', "restart_after_crash = on\n");
+$primary->start;
+$primary->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+$primary->backup('b');
+my $standby = PostgreSQL::Test::Cluster->new('standby');
+$standby->init_from_backup($primary, 'b', has_streaming => 1);
+$standby->start;
+
+for my $how ('immediate stop', 'backend killed')
+{
+       (my $f = "loser-$how.dat") =~ s/ /-/g;
+       my $a = $primary->background_psql('postgres');
+       $a->query_safe("BEGIN; SELECT test_fileops_create('$f', 384); SELECT 
test_fileops_mkdir('$f.d', 448);");
+       # make it durable and replicated
+       $primary->safe_psql('postgres', 'CREATE TABLE IF NOT EXISTS flush_me (a 
int)');
+       $primary->wait_for_catchup($standby);
+       ok(-e $standby->data_dir . "/$f", "$how: standby has the in-flight 
file");
+
+       if ($how eq 'immediate stop')
+       {
+               $primary->stop('immediate');
+               eval { $a->quit; };
+               $primary->start;
+       }
+       else
+       {
+               my $pid = $a->query_safe('SELECT pg_backend_pid()');
+               chomp $pid;
+               kill 'KILL', $pid;
+               eval { $a->quit; };
+               $primary->poll_query_until('postgres', 'SELECT 1', '1');
+       }
+       ok(!-e $primary->data_dir . "/$f", "$how: primary reversed the loser");
+       $primary->safe_psql('postgres', 'CHECKPOINT');
+       $primary->wait_for_catchup($standby);
+       ok(!-e $standby->data_dir . "/$f", "$how: standby reversed the loser 
file");
+       ok(!-e $standby->data_dir . "/$f.d", "$how: standby reversed the loser 
dir");
+
+}
+
+# The standby converges only when it is promoted and runs its own undo phase.
+$standby->promote;
+for my $how ('immediate-stop', 'backend-killed')
+{
+       ok(!-e $standby->data_dir . "/loser-$how.dat", "promoted standby: 
loser-$how.dat absent");
+}
+$standby->stop;
+$primary->stop;
+done_testing();
diff --git 
a/src/test/modules/test_fileops/t/028_fileops_recovery_apply_failure_count.pl 
b/src/test/modules/test_fileops/t/028_fileops_recovery_apply_failure_count.pl
new file mode 100644
index 00000000000..f3fa4f47b05
--- /dev/null
+++ 
b/src/test/modules/test_fileops/t/028_fileops_recovery_apply_failure_count.pl
@@ -0,0 +1,63 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# FILEOPS WRITE inverses vs. a real lstat() error (no injection point), with
+# no revert worker: each failing inverse is tried once by the startup undo
+# phase and counted in its final WARNING.
+#
+# Two crash losers overwrite three files (two in one transaction, one in the
+# other).  Before the restart the files' directory loses its search
+# permission, so all three inverses fail with EACCES.  The WARNING must say
+# 3 records, not 1 or 2: test_fileops/024 has a single failure, so it cannot
+# tell a count from a flag.
+#
+# Must not run as root (root ignores the directory permission).
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+plan skip_all => 'running as root: EACCES cannot be produced' if $> == 0;
+
+my $node = PostgreSQL::Test::Cluster->new('n');
+$node->init(allows_streaming => 1);
+$node->append_conf('postgresql.conf', "autovacuum = off\n");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+
+my $dir = PostgreSQL::Test::Utils::tempdir('eacces');
+for my $f ('w1', 'w2', 'w3')
+{
+       open(my $fh, '>', "$dir/$f.dat") or die "open: $!";
+       print $fh 'X' x 1024;
+       close($fh);
+}
+
+my $a = $node->background_psql('postgres', on_error_stop => 0);
+$a->query_safe("BEGIN; SELECT test_fileops_write('$dir/w1.dat', 0, 
'LOSER-A1');"
+         . " SELECT test_fileops_write('$dir/w2.dat', 0, 'LOSER-A2');");
+my $b = $node->background_psql('postgres', on_error_stop => 0);
+$b->query_safe("BEGIN; SELECT test_fileops_write('$dir/w3.dat', 0, 
'LOSER-B3');");
+$node->safe_psql('postgres', 'SELECT pg_switch_wal()');
+my $offset = -s $node->logfile;
+$node->stop('immediate');
+eval { $a->quit; };
+eval { $b->quit; };
+
+chmod(0600, $dir) or die "chmod $dir: $!";
+$node->start;
+chmod(0700, $dir) or die "chmod $dir: $!";
+
+my $log = slurp_file($node->logfile, $offset);
+my $warns = () = $log =~ /FILEOPS UNDO WRITE: could not stat/g;
+is($warns, 3, 'three inverses failed with a real EACCES');
+like($log,
+       qr/UNDO recovery complete: 2 transactions processed, 0 records applied, 
3 records could not be applied/,
+       'final WARNING counts 3 failed records');
+# Tried once and not retried: the bytes stay as the losers left them.
+is(substr(slurp_file("$dir/w1.dat"), 0, 8), 'LOSER-A1', 'w1 not restored');
+is(substr(slurp_file("$dir/w3.dat"), 0, 8), 'LOSER-B3', 'w3 not restored');
+note "RESULT stat_warnings=$warns";
+
+$node->stop;
+done_testing();
diff --git a/src/test/modules/test_fileops/t/029_fileops_insert_holdoff_gdb.pl 
b/src/test/modules/test_fileops/t/029_fileops_insert_holdoff_gdb.pl
new file mode 100644
index 00000000000..65658c5fa02
--- /dev/null
+++ b/src/test/modules/test_fileops/t/029_fileops_insert_holdoff_gdb.pl
@@ -0,0 +1,92 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# InsertXactUndoData() holds off a checkpoint's redo-pointer phase between
+# inserting an UNDO batch and publishing the chain head.  Make that interval
+# deterministic with gdb: stop the backend in PublishXactUndoChain(), after
+# the batch is in the WAL and before the head is visible, request a
+# CHECKPOINT, then kill the backend so the server goes through crash
+# recovery.  The loser's file must be removed.
+#
+# Without the hold-off, the checkpoint completes during the stop, its
+# XLOG_UNDO_INFLIGHT record lacks this chain, and the batch lies before the
+# redo pointer: nothing reverses the create.  Another commit flushes the
+# create's WAL record first, so this is outside the documented window.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use IPC::Run;
+use Time::HiRes qw(usleep);
+
+plan skip_all => 'gdb not found' unless -x '/usr/bin/gdb';
+
+my $node = PostgreSQL::Test::Cluster->new('n');
+$node->init(allows_streaming => 1);
+$node->append_conf('postgresql.conf',
+       "checkpoint_timeout = 1h\nrestart_after_crash = on\n");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+
+my $a = $node->background_psql('postgres');
+$a->query_safe('BEGIN; SELECT txid_current();');
+my $pid = $a->query_safe('SELECT pg_backend_pid()');
+chomp $pid;
+
+my $flag = "$PostgreSQL::Test::Utils::tmp_check/at_publish";
+my $cmds = "$PostgreSQL::Test::Utils::tmp_check/gdb.cmds";
+open(my $fh, '>', $cmds) or die;
+print $fh <<"EOF";
+set pagination off
+set confirm off
+break PublishXactUndoChain
+continue
+shell touch $flag
+shell sleep 6
+kill
+quit
+EOF
+close $fh;
+my ($gout, $gerr) = ('', '');
+my $gdb = IPC::Run::start([ '/usr/bin/gdb', '-q', '-batch', '-x', $cmds, '-p', 
$pid ],
+       '>', \$gout, '2>', \$gerr);
+sleep 2;    # let gdb attach and set the breakpoint
+
+$a->query_until(qr/go/, "\\echo go\nSELECT test_fileops_create('holdoff.dat', 
384);\n");
+for (1 .. 100) { last if -e $flag; usleep(100_000); }
+ok(-e $flag, 'backend stopped between batch insert and chain publish');
+
+# Make the create's WAL and UNDO records durable while the backend is still
+# stopped: another transaction's commit flushes everything inserted before
+# it.  Without this the records may still be in WAL buffers at the kill, and
+# the case would fall in the documented window.
+my $ins = $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()');
+$node->safe_psql('postgres', 'CREATE TABLE flush_me (a int)');
+my $durable = $node->safe_psql('postgres',
+       "SELECT pg_current_wal_flush_lsn() >= '$ins'::pg_lsn");
+is($durable, 't', 'the create is durable before the checkpoint');
+
+my $before = $node->safe_psql('postgres',
+       'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+my $c = $node->background_psql('postgres', on_error_stop => 0);
+$c->query_until(qr/go/, "\\echo go\nCHECKPOINT;\n");
+my $after = $before;
+for (1 .. 40)
+{
+       $after = $node->safe_psql('postgres',
+               'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+       last if $after ne $before;
+       usleep(100_000);
+}
+note 'checkpoint completed while the backend was stopped: '
+  . ($after ne $before ? 'yes' : 'no');
+
+$gdb->finish;
+note "gdb: $gout $gerr";
+eval { $a->quit; };
+eval { $c->quit; };
+$node->poll_query_until('postgres', 'SELECT 1', '1');
+ok(!-e $node->data_dir . '/holdoff.dat', 'crash recovery removed the loser 
file');
+
+$node->stop;
+done_testing();
FILEOPS invariant oracle for v10
================================

Invariant checked: after any mix of commit, rollback, crash, checkpoint,
2PC and standby replay, the files FILEOPS manages are exactly those the
committed transactions imply.

Scenarios: the full product of
  operation   create, mkdir, link, rename, write, delete   (via test_fileops)
  end         commit, rollback, rollback to savepoint, crash with the
              transaction in flight, crash after the abort record is
              flushed (injection point fileops-abort-after-flush),
              commit prepared, rollback prepared
  checkpoint  none, before the operation, between operation and end
  reuse       another transaction takes the freed name afterwards: no/yes
minus the combinations where no name is free to reuse: 186 scenarios.
Every scenario is checked on the primary right after it ends (non-crash
ends), and on the primary and the standby after one crash and again after
a second crash of the primary.

Usage:
  python3 oracle.py gen scenarios.json
  SCENARIOS=scenarios.json DUMP_DIR=out prove 910_oracle_runner.pl
  python3 oracle.py check scenarios.json out1 out2 out3

The model in oracle.py knows nothing about FILEOPS internals; the runner
only executes and dumps.  At most 6 aborts are paused per crash round
because the injection_points module has 8 wait slots.

Results on v10 (three runs, same 21 failures each time, none unstable)
---------------------------------------------------------------------
scenarios 186  runs 3  pass 165  fail 21  unstable 0
All 21 failures are crashes with the transaction in flight, seen on the
standby; the primary matches the model in all 186 scenarios.
FAIL s015 op=create end=crash_in_flight ckpt=none reuse=no
    after_crash/standby/s015_f: want -:None got f:
    after_second_crash/standby/s015_f: want -:None got f:
FAIL s017 op=create end=crash_in_flight ckpt=before reuse=no
    after_crash/standby/s017_f: want -:None got f:
    after_second_crash/standby/s017_f: want -:None got f:
FAIL s019 op=create end=crash_in_flight ckpt=between reuse=no
    after_crash/standby/s019_f: want -:None got f:
    after_second_crash/standby/s019_f: want -:None got f:
FAIL s051 op=mkdir end=crash_in_flight ckpt=none reuse=no
    after_crash/standby/s051_d: want -:None got d:None
    after_second_crash/standby/s051_d: want -:None got d:None
FAIL s053 op=mkdir end=crash_in_flight ckpt=before reuse=no
    after_crash/standby/s053_d: want -:None got d:None
    after_second_crash/standby/s053_d: want -:None got d:None
FAIL s055 op=mkdir end=crash_in_flight ckpt=between reuse=no
    after_crash/standby/s055_d: want -:None got d:None
    after_second_crash/standby/s055_d: want -:None got d:None
FAIL s087 op=link end=crash_in_flight ckpt=none reuse=no
    after_crash/standby/s087_l: want -:None got f:committed-original
    after_second_crash/standby/s087_l: want -:None got f:committed-original
FAIL s088 op=link end=crash_in_flight ckpt=none reuse=yes
    after_crash/standby/s088_src: want f:committed-original got 
f:reused-by-another-xact
    after_second_crash/standby/s088_src: want f:committed-original got 
f:reused-by-another-xact
FAIL s089 op=link end=crash_in_flight ckpt=before reuse=no
    after_crash/standby/s089_l: want -:None got f:committed-original
    after_second_crash/standby/s089_l: want -:None got f:committed-original
FAIL s090 op=link end=crash_in_flight ckpt=before reuse=yes
    after_crash/standby/s090_src: want f:committed-original got 
f:reused-by-another-xact
    after_second_crash/standby/s090_src: want f:committed-original got 
f:reused-by-another-xact
FAIL s091 op=link end=crash_in_flight ckpt=between reuse=no
    after_crash/standby/s091_l: want -:None got f:committed-original
    after_second_crash/standby/s091_l: want -:None got f:committed-original
FAIL s092 op=link end=crash_in_flight ckpt=between reuse=yes
    after_crash/standby/s092_src: want f:committed-original got 
f:reused-by-another-xact
    after_second_crash/standby/s092_src: want f:committed-original got 
f:reused-by-another-xact
FAIL s123 op=rename end=crash_in_flight ckpt=none reuse=no
    after_crash/standby/s123_a: want f:committed-original got -:None
    after_crash/standby/s123_b: want -:None got f:committed-original
    after_second_crash/standby/s123_a: want f:committed-original got -:None
    after_second_crash/standby/s123_b: want -:None got f:committed-original
FAIL s124 op=rename end=crash_in_flight ckpt=none reuse=yes
    after_crash/standby/s124_a: want f:committed-original got -:None
    after_second_crash/standby/s124_a: want f:committed-original got -:None
FAIL s125 op=rename end=crash_in_flight ckpt=before reuse=no
    after_crash/standby/s125_a: want f:committed-original got -:None
    after_crash/standby/s125_b: want -:None got f:committed-original
    after_second_crash/standby/s125_a: want f:committed-original got -:None
    after_second_crash/standby/s125_b: want -:None got f:committed-original
FAIL s126 op=rename end=crash_in_flight ckpt=before reuse=yes
    after_crash/standby/s126_a: want f:committed-original got -:None
    after_second_crash/standby/s126_a: want f:committed-original got -:None
FAIL s127 op=rename end=crash_in_flight ckpt=between reuse=no
    after_crash/standby/s127_a: want f:committed-original got -:None
    after_crash/standby/s127_b: want -:None got f:committed-original
    after_second_crash/standby/s127_a: want f:committed-original got -:None
    after_second_crash/standby/s127_b: want -:None got f:committed-original
FAIL s128 op=rename end=crash_in_flight ckpt=between reuse=yes
    after_crash/standby/s128_a: want f:committed-original got -:None
    after_second_crash/standby/s128_a: want f:committed-original got -:None
FAIL s153 op=write end=crash_in_flight ckpt=none reuse=no
    after_crash/standby/s153_w: want f:committed-original got 
f:NEW-BYTES-original
    after_second_crash/standby/s153_w: want f:committed-original got 
f:NEW-BYTES-original
FAIL s154 op=write end=crash_in_flight ckpt=before reuse=no
    after_crash/standby/s154_w: want f:committed-original got 
f:NEW-BYTES-original
    after_second_crash/standby/s154_w: want f:committed-original got 
f:NEW-BYTES-original
FAIL s155 op=write end=crash_in_flight ckpt=between reuse=no
    after_crash/standby/s155_w: want f:committed-original got 
f:NEW-BYTES-original
    after_second_crash/standby/s155_w: want f:committed-original got 
f:NEW-BYTES-original

===== oracle.py =====
"""Scenario generator, model and oracle for the FILEOPS invariant.

Invariant: after any mix of commit, rollback, crash, checkpoint, 2PC and
standby replay, the files FILEOPS manages are exactly those the committed
transactions imply.

    python3 -I oracle.py gen <scenarios.json>
    python3 -I oracle.py check <scenarios.json> <dump-dir> [<dump-dir> ...]

The model below knows nothing about FILEOPS internals: it computes the
expected state of each scenario's names from the history alone.  The runner
(t/910_oracle_runner.pl) executes the scenarios and dumps what is on disk;
`check` compares the two at every stage, on the primary and on the standby.
"""
import itertools
import json
import sys
from collections import defaultdict

OPS = ["create", "mkdir", "link", "rename", "write", "delete"]
ENDS = ["commit", "rollback", "rollback_to_savepoint", "crash_in_flight",
        "crash_after_abort_flush", "commit_prepared", "rollback_prepared"]
CKPTS = ["none", "before", "between"]
REUSE = ["no", "yes"]

CRASH_ENDS = {"crash_in_flight", "crash_after_abort_flush"}
COMMITTED_ENDS = {"commit", "commit_prepared"}

SETUP_CONTENT = "committed-original"
WRITE_CONTENT = "NEW-BYTES"
REUSE_CONTENT = "reused-by-another-xact"


def reuse_target(op, committed):
    """The name that is free after the end, for another transaction to take."""
    if committed:
        return None
    return {"create": "f", "mkdir": "d", "link": "l", "rename": "b"}.get(op)


def scenarios():
    out = []
    for i, (op, end, ckpt, reuse) in enumerate(itertools.product(OPS, ENDS, 
CKPTS, REUSE)):
        committed = end in COMMITTED_ENDS
        if reuse == "yes" and reuse_target(op, committed) is None:
            continue                       # nothing is free to reuse
        out.append({"id": f"s{len(out):03d}", "op": op, "end": end, "ckpt": 
ckpt,
                    "reuse": reuse, "reuse_name": reuse_target(op, committed)})
    return out


def model(s):
    """Expected state of the scenario's names: {name: (type, content)}.
    type is 'f' (file), 'd' (directory) or '-' (absent)."""
    committed = s["end"] in COMMITTED_ENDS
    n = lambda x: f"{s['id']}_{x}"
    st = {}
    op = s["op"]
    if op == "create":
        st[n("f")] = ("f", "") if committed else ("-", None)
    elif op == "mkdir":
        st[n("d")] = ("d", None) if committed else ("-", None)
    elif op == "link":
        st[n("src")] = ("f", SETUP_CONTENT)
        st[n("l")] = ("f", SETUP_CONTENT) if committed else ("-", None)
    elif op == "rename":
        st[n("a")] = ("-", None) if committed else ("f", SETUP_CONTENT)
        st[n("b")] = ("f", SETUP_CONTENT) if committed else ("-", None)
    elif op == "write":
        st[n("w")] = ("f", WRITE_CONTENT + SETUP_CONTENT[len(WRITE_CONTENT):]
                      if committed else SETUP_CONTENT)
    elif op == "delete":
        st[n("x")] = ("-", None) if committed else ("f", SETUP_CONTENT)
    if s["reuse"] == "yes":
        name = n(s["reuse_name"])
        st[name] = ("d", None) if op == "mkdir" else ("f", REUSE_CONTENT)
    return st


def stages_for(s):
    """Stages at which the scenario's state is defined.  'online' is the
    primary right after the end; crash scenarios only exist after the crash."""
    first = [] if s["end"] in CRASH_ENDS else [("online", "primary")]
    return first + [(st, node) for st in ("after_crash", "after_second_crash")
                    for node in ("primary", "standby")]


def read_dump(path):
    """Lines: <stage> <node> <name> <type> <content-or-dash>"""
    d = defaultdict(dict)
    for line in open(path):
        stage, node, name, typ, content = line.rstrip("\n").split("\t")
        d[(stage, node)][name] = (typ, None if content == "\\N" else content)
    return d


def check(spec, dumps):
    scen = json.load(open(spec))
    runs = [read_dump(f"{d}/dump.tsv") for d in dumps]
    result = {}
    for s in scen:
        want = model(s)
        verdicts = []
        for run in runs:
            bad = []
            for stage, node in stages_for(s):
                got = run.get((stage, node), {})
                for name, w in want.items():
                    g = got.get(name, ("-", None))
                    if (g[0], g[1] if w[0] == "f" else None) != (w[0], w[1] if 
w[0] == "f" else None):
                        bad.append(f"{stage}/{node}/{name}: want {w[0]}:{w[1]} 
got {g[0]}:{g[1]}")
            verdicts.append(bad)
        result[s["id"]] = (s, verdicts)
    stable_fail, flaky, ok = [], [], []
    for sid, (s, v) in result.items():
        fails = [bool(b) for b in v]
        if all(fails):
            stable_fail.append((s, v[0]))
        elif any(fails):
            flaky.append((s, [b for b in v if b][0]))
        else:
            ok.append(s)
    print(f"scenarios {len(result)}  runs {len(runs)}  pass {len(ok)}  "
          f"fail {len(stable_fail)}  unstable {len(flaky)}")
    for label, group in (("FAIL", stable_fail), ("UNSTABLE", flaky)):
        for s, bad in group:
            print(f"{label} {s['id']} op={s['op']} end={s['end']} 
ckpt={s['ckpt']} reuse={s['reuse']}")
            for b in bad[:4]:
                print(f"    {b}")


if __name__ == "__main__":
    if sys.argv[1] == "gen":
        sc = scenarios()
        json.dump(sc, open(sys.argv[2], "w"), indent=1)
        print(f"{len(sc)} scenarios")
    elif sys.argv[1] == "check":
        check(sys.argv[2], sys.argv[3:])

===== 910_oracle_runner.pl =====
# Runner for the FILEOPS invariant oracle (oracle.py).  Executes every
# scenario in SCENARIOS (a JSON file from `oracle.py gen`) on a primary with
# a streaming standby, and writes what is on disk for each scenario's names to
# $ENV{DUMP_DIR}/dump.tsv at three stages:
#   online              the primary right after each non-crash scenario ends
#   after_crash         primary and standby after one crash of the primary
#   after_second_crash  both again after a second crash (replays the same WAL)
# The runner does not judge anything; oracle.py compares against its model.
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use JSON::PP;
use Time::HiRes qw(usleep);

my $scen = decode_json(slurp_file($ENV{SCENARIOS}));
my $dumpdir = $ENV{DUMP_DIR} or die 'DUMP_DIR';
open(my $dump, '>', "$dumpdir/dump.tsv") or die;

my $SETUP = 'committed-original';
my $NEW = 'NEW-BYTES';
my $REUSE = 'reused-by-another-xact';

my $primary = PostgreSQL::Test::Cluster->new('primary');
$primary->init(allows_streaming => 1);
$primary->append_conf('postgresql.conf', qq{
max_connections = 250
max_prepared_transactions = 100
checkpoint_timeout = 1h
autovacuum = off
});
$primary->start;
$primary->safe_psql('postgres',
        'CREATE EXTENSION test_fileops; CREATE EXTENSION injection_points');
$primary->backup('b');
my $standby = PostgreSQL::Test::Cluster->new('standby');
$standby->init_from_backup($primary, 'b', has_streaming => 1);
$standby->start;

sub sql { $primary->safe_psql('postgres', $_[0]) }
sub nm { my ($s, $x) = @_; return "$s->{id}_$x" }

sub mkfile
{
        my ($name, $content) = @_;
        sql("BEGIN; SELECT test_fileops_create('$name', 384);"
                  . " SELECT test_fileops_write('$name', 0, '$content'); 
COMMIT;");
}

sub setup
{
        my ($s) = @_;
        my %fixture = (link => 'src', rename => 'a', write => 'w', delete => 
'x');
        mkfile(nm($s, $fixture{ $s->{op} }), $SETUP) if $fixture{ $s->{op} };
}

sub op_sql
{
        my ($s) = @_;
        my $o = $s->{op};
        return "SELECT test_fileops_create('" . nm($s, 'f') . "', 384)" if $o 
eq 'create';
        return "SELECT test_fileops_mkdir('" . nm($s, 'd') . "', 448)" if $o eq 
'mkdir';
        return "SELECT test_fileops_link('" . nm($s, 'src') . "', '" . nm($s, 
'l') . "')" if $o eq 'link';
        return "SELECT test_fileops_rename('" . nm($s, 'a') . "', '" . nm($s, 
'b') . "')" if $o eq 'rename';
        return "SELECT test_fileops_write('" . nm($s, 'w') . "', 0, '$NEW')" if 
$o eq 'write';
        return "SELECT test_fileops_delete('" . nm($s, 'x') . "')" if $o eq 
'delete';
        die $o;
}

sub reuse
{
        my ($s) = @_;
        return unless $s->{reuse} eq 'yes';
        my $name = nm($s, $s->{reuse_name});
        if ($s->{op} eq 'mkdir') { sql("SELECT test_fileops_mkdir('$name', 
448)") }
        else { mkfile($name, $REUSE) }
}

sub start_xact
{
        my ($s) = @_;
        sql('CHECKPOINT') if $s->{ckpt} eq 'before';
        my $bg = $primary->background_psql('postgres', on_error_stop => 0);
        $bg->query_safe('BEGIN');
        $bg->query_safe('SAVEPOINT sp') if $s->{end} eq 'rollback_to_savepoint';
        $bg->query_safe(op_sql($s));
        sql('CHECKPOINT') if $s->{ckpt} eq 'between';
        return $bg;
}

sub dump_state
{
        my ($stage, $node) = @_;
        my $dir = $node->data_dir;
        opendir(my $d, $dir) or die;
        for my $e (sort grep { /^s\d{3}_/ } readdir($d))
        {
                my $p = "$dir/$e";
                my ($t, $c) = -d $p ? ('d', '\N') : ('f', slurp_file($p));
                print $dump join("\t", $stage, $node->name, $e, $t, $c), "\n";
        }
        closedir($d);
}

my @crash = grep { $_->{end} =~ /^crash_/ } @$scen;
my @plain = grep { $_->{end} !~ /^crash_/ } @$scen;
setup($_) for @$scen;

# Non-crash scenarios, one at a time.
for my $s (@plain)
{
        my $bg = start_xact($s);
        my $e = $s->{end};
        if ($e eq 'commit') { $bg->query_safe('COMMIT') }
        elsif ($e eq 'rollback') { $bg->query_safe('ROLLBACK') }
        elsif ($e eq 'rollback_to_savepoint') { $bg->query_safe('ROLLBACK TO 
sp; COMMIT') }
        else
        {
                $bg->query_safe("PREPARE TRANSACTION 'g$s->{id}'");
                sql(($e eq 'commit_prepared' ? 'COMMIT' : 'ROLLBACK') . " 
PREPARED 'g$s->{id}'");
        }
        $bg->quit;
        reuse($s);
}
dump_state('online', $primary);

# Crash scenarios, in rounds that each end with one crash.  The injection
# point module has 8 wait slots, so at most 6 aborts are paused per round;
# the in-flight losers all go in the first round.
my @after = grep { $_->{end} eq 'crash_after_abort_flush' } @crash;
my @inflight = grep { $_->{end} eq 'crash_in_flight' } @crash;
my @rounds = ([ @inflight, splice(@after, 0, 6) ]);
push @rounds, [ splice(@after, 0, 6) ] while @after;
for my $round (@rounds)
{
        my @open = map { [ $_, start_xact($_) ] } @$round;
        sql("SELECT injection_points_attach('fileops-abort-after-flush', 
'wait')");
        my $waiting = 0;
        for my $o (@open)
        {
                next unless $o->[0]{end} eq 'crash_after_abort_flush';
                $o->[1]->query_until(qr/go/, "\\echo go\nROLLBACK;\n");
                # A deferred delete has no inverse, so its abort does not pause.
                $waiting++ unless $o->[0]{op} eq 'delete';
        }
        $primary->poll_query_until('postgres',
                "SELECT count(*) = $waiting FROM pg_stat_activity WHERE 
wait_event = 'fileops-abort-after-flush'")
          or die 'aborts did not pause';
        note "crash round: " . scalar(@open) . " scenarios, paused after abort 
flush: $waiting";
        $primary->stop('immediate');
        eval { $_->[1]->quit } for @open;
        $primary->start;
        reuse($_) for @$round;
}
$primary->wait_for_catchup($standby);
dump_state('after_crash', $_) for ($primary, $standby);

$primary->stop('immediate');
$primary->start;
sql('SELECT 1');
$primary->wait_for_catchup($standby);
dump_state('after_second_crash', $_) for ($primary, $standby);

close($dump);
ok(1, 'runner finished');
$standby->stop;
$primary->stop;
done_testing();

Reply via email to