Hi Greg,
The four fixes hold: your tests pass and each fails when I revert its
fix. Two cases outside the window you document still end with files
that differ from what the committed transactions imply; the attached
oracle runs 186 scenarios.
On Wed, Nov 11, 2026 at 1:09 PM Greg Burd <[email protected]> wrote:
> The abort record carries the transaction's FILEOPS inverses, and abort
> redo applies them, so a standby, and the primary after a crash, end up
> with the files the primary had after ROLLBACK
Not for a crash loser: the primary's startup undo phase removes its
files, but a crash writes no abort record, so a streaming standby keeps
them until it is promoted. I see it for create, mkdir, link, rename and
write, and on v9 for the two I tried there. After a link, if another
transaction then creates that name, the standby's source file ends up
with that transaction's bytes.
FileOpsCreate() opens without O_EXCL and writes the create UNDO even if
the file existed, so a rollback unlinks a committed file, on the
primary, the standby and after a crash. No in-core caller does that;
the first diff makes it an ERROR, as FileOpsMkdir() already does.
The hold-off in InsertXactUndoData() and ROLLBACK PREPARED keeping
DELAY_CHKPT_START have no test that fails without them. The second
diff adds one for each (the hold-off one drives gdb, so take it as a
repro rather than a test for the tree), the crash-loser repro, and a
copy of 024 with three failures.
> Perhaps I should separate them into a 4th patch to make that explicit.
In 0001 the subtransaction and 2PC functions are about 140 of the 1,626
lines gcov counts; at most 58 lines run only under the tests using them.
Manu
diff --git a/src/backend/storage/file/README b/src/backend/storage/file/README
index 26c9a5b5120..ed29ae8bbee 100644
--- a/src/backend/storage/file/README
+++ b/src/backend/storage/file/README
@@ -66,7 +66,8 @@ These run when the API is called, so the file or directory
can be used
within the same transaction, and each registers an abort-time undo action
that reverses it if the transaction rolls back:
- FileOpsCreate -- create() now; UNDO/pending-delete on abort
+ FileOpsCreate -- create() now, an ERROR if the file exists;
UNDO/pending-delete
+ on abort
FileOpsMkdir -- mkdir() now; UNDO rmdir on abort
FileOpsMkdirTree -- mkdir() now; UNDO removes the whole tree on abort
FileOpsChmod -- chmod() now; UNDO restore old mode on abort
diff --git a/src/backend/storage/file/fileops.c
b/src/backend/storage/file/fileops.c
index 386eb618910..8ad7ab281ff 100644
--- a/src/backend/storage/file/fileops.c
+++ b/src/backend/storage/file/fileops.c
@@ -431,6 +431,11 @@ FileOpsCancelPendingDelete(const char *path, bool
at_commit)
* and logs the creation to WAL. If register_delete is true, the file will
* be deleted if the transaction aborts.
*
+ * A path that already exists is an ERROR (ERRCODE_DUPLICATE_FILE), as for
+ * FileOpsMkdir(), FileOpsLink() and FileOpsRename(). The UNDO record
+ * unlinks the path, so the path must name a file this call created: opening
+ * an existing file here would let a rollback remove a committed one.
+ *
* Platform handling:
* - Linux/FreeBSD: O_DIRECT passed directly to open()
* - macOS: F_NOCACHE fcntl applied after open()
@@ -446,7 +451,7 @@ FileOpsCreate(const char *path, int flags, mode_t mode,
bool register_delete)
Assert(!IsInParallelMode());
- fd = OpenTransientFilePerm(path, flags | O_CREAT, mode);
+ fd = OpenTransientFilePerm(path, flags | O_CREAT | O_EXCL, mode);
if (fd < 0)
ereport(ERROR,
(errcode_for_file_access(),
diff --git a/src/test/modules/test_fileops/meson.build
b/src/test/modules/test_fileops/meson.build
index e14676912a4..e9c51446bed 100644
--- a/src/test/modules/test_fileops/meson.build
+++ b/src/test/modules/test_fileops/meson.build
@@ -44,6 +44,7 @@ tests += {
't/018_fileops_rename.pl',
't/023_fileops_inline_abort_ctx.pl',
't/024_fileops_recovery_apply_failure.pl',
+ 't/025_fileops_create_existing.pl',
],
},
}
diff --git a/src/test/modules/test_fileops/t/025_fileops_create_existing.pl
b/src/test/modules/test_fileops/t/025_fileops_create_existing.pl
new file mode 100644
index 00000000000..a0db2dc29f0
--- /dev/null
+++ b/src/test/modules/test_fileops/t/025_fileops_create_existing.pl
@@ -0,0 +1,56 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# FileOpsCreate() on a path that already exists, then ROLLBACK. The UNDO
+# record of a create unlinks the path, so if the call opened an existing file
+# a rollback would remove a committed one. The committed file must survive
+# on the primary, on a standby replaying the abort record, and on the primary
+# after crash recovery replays it.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+$primary->start;
+$primary->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+$primary->backup('backup');
+my $standby = PostgreSQL::Test::Cluster->new('standby');
+$standby->init_from_backup($primary, 'backup', has_streaming => 1);
+$standby->start;
+
+$primary->safe_psql('postgres', q{
+BEGIN;
+SELECT test_fileops_create('existing.dat', 384);
+SELECT test_fileops_write('existing.dat', 0, 'committed');
+COMMIT;
+});
+
+my ($ret, $out, $err) = $primary->psql('postgres', q{
+BEGIN;
+SELECT test_fileops_create('existing.dat', 384);
+ROLLBACK;
+});
+like($err, qr/could not create file "existing.dat": File exists/,
+ 'creating an existing path is an ERROR');
+
+$primary->wait_for_catchup($standby);
+for my $stage ('online', 'after crash')
+{
+ if ($stage eq 'after crash')
+ {
+ $primary->stop('immediate');
+ $primary->start;
+ }
+ my $f = $primary->data_dir . '/existing.dat';
+ is(-e $f ? slurp_file($f) : '(missing)',
+ 'committed', "$stage: primary keeps the committed file");
+}
+my $sf = $standby->data_dir . '/existing.dat';
+is(-e $sf ? slurp_file($sf) : '(missing)',
+ 'committed', 'standby keeps the committed file');
+
+$standby->stop;
+$primary->stop;
+done_testing();
diff --git a/src/test/modules/test_fileops/meson.build
b/src/test/modules/test_fileops/meson.build
index e14676912a4..f6e4d7fc69c 100644
--- a/src/test/modules/test_fileops/meson.build
+++ b/src/test/modules/test_fileops/meson.build
@@ -44,6 +44,10 @@ tests += {
't/018_fileops_rename.pl',
't/023_fileops_inline_abort_ctx.pl',
't/024_fileops_recovery_apply_failure.pl',
+ 't/026_fileops_abort_checkpoint_order.pl',
+ 't/027_fileops_crash_loser_standby.pl',
+ 't/028_fileops_recovery_apply_failure_count.pl',
+ 't/029_fileops_insert_holdoff_gdb.pl',
],
},
}
diff --git
a/src/test/modules/test_fileops/t/026_fileops_abort_checkpoint_order.pl
b/src/test/modules/test_fileops/t/026_fileops_abort_checkpoint_order.pl
new file mode 100644
index 00000000000..0e68ae43398
--- /dev/null
+++ b/src/test/modules/test_fileops/t/026_fileops_abort_checkpoint_order.pl
@@ -0,0 +1,75 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# A checkpoint must not move the redo pointer past an abort record whose
+# FILEOPS inverses are not yet durable. Pause the aborting backend after the
+# abort record is flushed and before it applies the inverses, request a
+# CHECKPOINT, then crash. Recovery must replay the abort record and remove
+# the file, for ROLLBACK and for ROLLBACK PREPARED.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+plan skip_all => 'injection points not supported by this build'
+ unless $ENV{enable_injection_points} eq 'yes';
+
+my $node = PostgreSQL::Test::Cluster->new('n');
+$node->init(allows_streaming => 1);
+$node->append_conf('postgresql.conf',
+ "max_prepared_transactions = 5\ncheckpoint_timeout = 1h\n");
+$node->start;
+$node->safe_psql('postgres',
+ 'CREATE EXTENSION test_fileops; CREATE EXTENSION injection_points');
+
+for my $case ('rollback', 'rollback prepared')
+{
+ (my $f = "ck-$case.dat") =~ s/ /-/g;
+ $node->safe_psql('postgres',
+ "SELECT injection_points_attach('fileops-abort-after-flush',
'wait')");
+ my $before = $node->safe_psql('postgres',
+ 'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+
+ my $a = $node->background_psql('postgres');
+ if ($case eq 'rollback')
+ {
+ $a->query_until(qr/go/, qq{\\echo go
+BEGIN; SELECT test_fileops_create('$f', 384); ROLLBACK;
+});
+ }
+ else
+ {
+ $node->safe_psql('postgres',
+ "BEGIN; SELECT test_fileops_create('$f', 384); PREPARE
TRANSACTION 'p';");
+ $a->query_until(qr/go/, qq{\\echo go
+ROLLBACK PREPARED 'p';
+});
+ }
+ $node->wait_for_event('client backend', 'fileops-abort-after-flush');
+ ok(-e $node->data_dir . "/$f", "$case: file still there at the pause");
+
+ my $c = $node->background_psql('postgres');
+ $c->query_until(qr/go/, "\\echo go\nCHECKPOINT;\n");
+
+ # Give the checkpoint time to finish if nothing holds it back.
+ my $after = $before;
+ for (1 .. 50)
+ {
+ $after = $node->safe_psql('postgres',
+ 'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+ last if $after ne $before;
+ usleep(100_000);
+ }
+ note "$case: checkpoint completed during the pause: "
+ . ($after ne $before ? 'yes' : 'no');
+
+ $node->stop('immediate');
+ eval { $a->quit; };
+ eval { $c->quit; };
+ $node->start;
+ ok(!-e $node->data_dir . "/$f", "$case: crash recovery removed the
file");
+}
+
+$node->stop;
+done_testing();
diff --git a/src/test/modules/test_fileops/t/027_fileops_crash_loser_standby.pl
b/src/test/modules/test_fileops/t/027_fileops_crash_loser_standby.pl
new file mode 100644
index 00000000000..04d77a3f360
--- /dev/null
+++ b/src/test/modules/test_fileops/t/027_fileops_crash_loser_standby.pl
@@ -0,0 +1,63 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# A transaction is in flight with a FILEOPS create when the primary crashes.
+# The primary's startup undo phase reverses it. A crashed transaction leaves
+# no abort record, so the question is whether the standby, replaying the
+# primary's WAL, ends with the same files.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 1);
+$primary->append_conf('postgresql.conf', "restart_after_crash = on\n");
+$primary->start;
+$primary->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+$primary->backup('b');
+my $standby = PostgreSQL::Test::Cluster->new('standby');
+$standby->init_from_backup($primary, 'b', has_streaming => 1);
+$standby->start;
+
+for my $how ('immediate stop', 'backend killed')
+{
+ (my $f = "loser-$how.dat") =~ s/ /-/g;
+ my $a = $primary->background_psql('postgres');
+ $a->query_safe("BEGIN; SELECT test_fileops_create('$f', 384); SELECT
test_fileops_mkdir('$f.d', 448);");
+ # make it durable and replicated
+ $primary->safe_psql('postgres', 'CREATE TABLE IF NOT EXISTS flush_me (a
int)');
+ $primary->wait_for_catchup($standby);
+ ok(-e $standby->data_dir . "/$f", "$how: standby has the in-flight
file");
+
+ if ($how eq 'immediate stop')
+ {
+ $primary->stop('immediate');
+ eval { $a->quit; };
+ $primary->start;
+ }
+ else
+ {
+ my $pid = $a->query_safe('SELECT pg_backend_pid()');
+ chomp $pid;
+ kill 'KILL', $pid;
+ eval { $a->quit; };
+ $primary->poll_query_until('postgres', 'SELECT 1', '1');
+ }
+ ok(!-e $primary->data_dir . "/$f", "$how: primary reversed the loser");
+ $primary->safe_psql('postgres', 'CHECKPOINT');
+ $primary->wait_for_catchup($standby);
+ ok(!-e $standby->data_dir . "/$f", "$how: standby reversed the loser
file");
+ ok(!-e $standby->data_dir . "/$f.d", "$how: standby reversed the loser
dir");
+
+}
+
+# The standby converges only when it is promoted and runs its own undo phase.
+$standby->promote;
+for my $how ('immediate-stop', 'backend-killed')
+{
+ ok(!-e $standby->data_dir . "/loser-$how.dat", "promoted standby:
loser-$how.dat absent");
+}
+$standby->stop;
+$primary->stop;
+done_testing();
diff --git
a/src/test/modules/test_fileops/t/028_fileops_recovery_apply_failure_count.pl
b/src/test/modules/test_fileops/t/028_fileops_recovery_apply_failure_count.pl
new file mode 100644
index 00000000000..f3fa4f47b05
--- /dev/null
+++
b/src/test/modules/test_fileops/t/028_fileops_recovery_apply_failure_count.pl
@@ -0,0 +1,63 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# FILEOPS WRITE inverses vs. a real lstat() error (no injection point), with
+# no revert worker: each failing inverse is tried once by the startup undo
+# phase and counted in its final WARNING.
+#
+# Two crash losers overwrite three files (two in one transaction, one in the
+# other). Before the restart the files' directory loses its search
+# permission, so all three inverses fail with EACCES. The WARNING must say
+# 3 records, not 1 or 2: test_fileops/024 has a single failure, so it cannot
+# tell a count from a flag.
+#
+# Must not run as root (root ignores the directory permission).
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+plan skip_all => 'running as root: EACCES cannot be produced' if $> == 0;
+
+my $node = PostgreSQL::Test::Cluster->new('n');
+$node->init(allows_streaming => 1);
+$node->append_conf('postgresql.conf', "autovacuum = off\n");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+
+my $dir = PostgreSQL::Test::Utils::tempdir('eacces');
+for my $f ('w1', 'w2', 'w3')
+{
+ open(my $fh, '>', "$dir/$f.dat") or die "open: $!";
+ print $fh 'X' x 1024;
+ close($fh);
+}
+
+my $a = $node->background_psql('postgres', on_error_stop => 0);
+$a->query_safe("BEGIN; SELECT test_fileops_write('$dir/w1.dat', 0,
'LOSER-A1');"
+ . " SELECT test_fileops_write('$dir/w2.dat', 0, 'LOSER-A2');");
+my $b = $node->background_psql('postgres', on_error_stop => 0);
+$b->query_safe("BEGIN; SELECT test_fileops_write('$dir/w3.dat', 0,
'LOSER-B3');");
+$node->safe_psql('postgres', 'SELECT pg_switch_wal()');
+my $offset = -s $node->logfile;
+$node->stop('immediate');
+eval { $a->quit; };
+eval { $b->quit; };
+
+chmod(0600, $dir) or die "chmod $dir: $!";
+$node->start;
+chmod(0700, $dir) or die "chmod $dir: $!";
+
+my $log = slurp_file($node->logfile, $offset);
+my $warns = () = $log =~ /FILEOPS UNDO WRITE: could not stat/g;
+is($warns, 3, 'three inverses failed with a real EACCES');
+like($log,
+ qr/UNDO recovery complete: 2 transactions processed, 0 records applied,
3 records could not be applied/,
+ 'final WARNING counts 3 failed records');
+# Tried once and not retried: the bytes stay as the losers left them.
+is(substr(slurp_file("$dir/w1.dat"), 0, 8), 'LOSER-A1', 'w1 not restored');
+is(substr(slurp_file("$dir/w3.dat"), 0, 8), 'LOSER-B3', 'w3 not restored');
+note "RESULT stat_warnings=$warns";
+
+$node->stop;
+done_testing();
diff --git a/src/test/modules/test_fileops/t/029_fileops_insert_holdoff_gdb.pl
b/src/test/modules/test_fileops/t/029_fileops_insert_holdoff_gdb.pl
new file mode 100644
index 00000000000..65658c5fa02
--- /dev/null
+++ b/src/test/modules/test_fileops/t/029_fileops_insert_holdoff_gdb.pl
@@ -0,0 +1,92 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# InsertXactUndoData() holds off a checkpoint's redo-pointer phase between
+# inserting an UNDO batch and publishing the chain head. Make that interval
+# deterministic with gdb: stop the backend in PublishXactUndoChain(), after
+# the batch is in the WAL and before the head is visible, request a
+# CHECKPOINT, then kill the backend so the server goes through crash
+# recovery. The loser's file must be removed.
+#
+# Without the hold-off, the checkpoint completes during the stop, its
+# XLOG_UNDO_INFLIGHT record lacks this chain, and the batch lies before the
+# redo pointer: nothing reverses the create. Another commit flushes the
+# create's WAL record first, so this is outside the documented window.
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+use IPC::Run;
+use Time::HiRes qw(usleep);
+
+plan skip_all => 'gdb not found' unless -x '/usr/bin/gdb';
+
+my $node = PostgreSQL::Test::Cluster->new('n');
+$node->init(allows_streaming => 1);
+$node->append_conf('postgresql.conf',
+ "checkpoint_timeout = 1h\nrestart_after_crash = on\n");
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_fileops');
+
+my $a = $node->background_psql('postgres');
+$a->query_safe('BEGIN; SELECT txid_current();');
+my $pid = $a->query_safe('SELECT pg_backend_pid()');
+chomp $pid;
+
+my $flag = "$PostgreSQL::Test::Utils::tmp_check/at_publish";
+my $cmds = "$PostgreSQL::Test::Utils::tmp_check/gdb.cmds";
+open(my $fh, '>', $cmds) or die;
+print $fh <<"EOF";
+set pagination off
+set confirm off
+break PublishXactUndoChain
+continue
+shell touch $flag
+shell sleep 6
+kill
+quit
+EOF
+close $fh;
+my ($gout, $gerr) = ('', '');
+my $gdb = IPC::Run::start([ '/usr/bin/gdb', '-q', '-batch', '-x', $cmds, '-p',
$pid ],
+ '>', \$gout, '2>', \$gerr);
+sleep 2; # let gdb attach and set the breakpoint
+
+$a->query_until(qr/go/, "\\echo go\nSELECT test_fileops_create('holdoff.dat',
384);\n");
+for (1 .. 100) { last if -e $flag; usleep(100_000); }
+ok(-e $flag, 'backend stopped between batch insert and chain publish');
+
+# Make the create's WAL and UNDO records durable while the backend is still
+# stopped: another transaction's commit flushes everything inserted before
+# it. Without this the records may still be in WAL buffers at the kill, and
+# the case would fall in the documented window.
+my $ins = $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()');
+$node->safe_psql('postgres', 'CREATE TABLE flush_me (a int)');
+my $durable = $node->safe_psql('postgres',
+ "SELECT pg_current_wal_flush_lsn() >= '$ins'::pg_lsn");
+is($durable, 't', 'the create is durable before the checkpoint');
+
+my $before = $node->safe_psql('postgres',
+ 'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+my $c = $node->background_psql('postgres', on_error_stop => 0);
+$c->query_until(qr/go/, "\\echo go\nCHECKPOINT;\n");
+my $after = $before;
+for (1 .. 40)
+{
+ $after = $node->safe_psql('postgres',
+ 'SELECT checkpoint_lsn FROM pg_control_checkpoint()');
+ last if $after ne $before;
+ usleep(100_000);
+}
+note 'checkpoint completed while the backend was stopped: '
+ . ($after ne $before ? 'yes' : 'no');
+
+$gdb->finish;
+note "gdb: $gout $gerr";
+eval { $a->quit; };
+eval { $c->quit; };
+$node->poll_query_until('postgres', 'SELECT 1', '1');
+ok(!-e $node->data_dir . '/holdoff.dat', 'crash recovery removed the loser
file');
+
+$node->stop;
+done_testing();
FILEOPS invariant oracle for v10
================================
Invariant checked: after any mix of commit, rollback, crash, checkpoint,
2PC and standby replay, the files FILEOPS manages are exactly those the
committed transactions imply.
Scenarios: the full product of
operation create, mkdir, link, rename, write, delete (via test_fileops)
end commit, rollback, rollback to savepoint, crash with the
transaction in flight, crash after the abort record is
flushed (injection point fileops-abort-after-flush),
commit prepared, rollback prepared
checkpoint none, before the operation, between operation and end
reuse another transaction takes the freed name afterwards: no/yes
minus the combinations where no name is free to reuse: 186 scenarios.
Every scenario is checked on the primary right after it ends (non-crash
ends), and on the primary and the standby after one crash and again after
a second crash of the primary.
Usage:
python3 oracle.py gen scenarios.json
SCENARIOS=scenarios.json DUMP_DIR=out prove 910_oracle_runner.pl
python3 oracle.py check scenarios.json out1 out2 out3
The model in oracle.py knows nothing about FILEOPS internals; the runner
only executes and dumps. At most 6 aborts are paused per crash round
because the injection_points module has 8 wait slots.
Results on v10 (three runs, same 21 failures each time, none unstable)
---------------------------------------------------------------------
scenarios 186 runs 3 pass 165 fail 21 unstable 0
All 21 failures are crashes with the transaction in flight, seen on the
standby; the primary matches the model in all 186 scenarios.
FAIL s015 op=create end=crash_in_flight ckpt=none reuse=no
after_crash/standby/s015_f: want -:None got f:
after_second_crash/standby/s015_f: want -:None got f:
FAIL s017 op=create end=crash_in_flight ckpt=before reuse=no
after_crash/standby/s017_f: want -:None got f:
after_second_crash/standby/s017_f: want -:None got f:
FAIL s019 op=create end=crash_in_flight ckpt=between reuse=no
after_crash/standby/s019_f: want -:None got f:
after_second_crash/standby/s019_f: want -:None got f:
FAIL s051 op=mkdir end=crash_in_flight ckpt=none reuse=no
after_crash/standby/s051_d: want -:None got d:None
after_second_crash/standby/s051_d: want -:None got d:None
FAIL s053 op=mkdir end=crash_in_flight ckpt=before reuse=no
after_crash/standby/s053_d: want -:None got d:None
after_second_crash/standby/s053_d: want -:None got d:None
FAIL s055 op=mkdir end=crash_in_flight ckpt=between reuse=no
after_crash/standby/s055_d: want -:None got d:None
after_second_crash/standby/s055_d: want -:None got d:None
FAIL s087 op=link end=crash_in_flight ckpt=none reuse=no
after_crash/standby/s087_l: want -:None got f:committed-original
after_second_crash/standby/s087_l: want -:None got f:committed-original
FAIL s088 op=link end=crash_in_flight ckpt=none reuse=yes
after_crash/standby/s088_src: want f:committed-original got
f:reused-by-another-xact
after_second_crash/standby/s088_src: want f:committed-original got
f:reused-by-another-xact
FAIL s089 op=link end=crash_in_flight ckpt=before reuse=no
after_crash/standby/s089_l: want -:None got f:committed-original
after_second_crash/standby/s089_l: want -:None got f:committed-original
FAIL s090 op=link end=crash_in_flight ckpt=before reuse=yes
after_crash/standby/s090_src: want f:committed-original got
f:reused-by-another-xact
after_second_crash/standby/s090_src: want f:committed-original got
f:reused-by-another-xact
FAIL s091 op=link end=crash_in_flight ckpt=between reuse=no
after_crash/standby/s091_l: want -:None got f:committed-original
after_second_crash/standby/s091_l: want -:None got f:committed-original
FAIL s092 op=link end=crash_in_flight ckpt=between reuse=yes
after_crash/standby/s092_src: want f:committed-original got
f:reused-by-another-xact
after_second_crash/standby/s092_src: want f:committed-original got
f:reused-by-another-xact
FAIL s123 op=rename end=crash_in_flight ckpt=none reuse=no
after_crash/standby/s123_a: want f:committed-original got -:None
after_crash/standby/s123_b: want -:None got f:committed-original
after_second_crash/standby/s123_a: want f:committed-original got -:None
after_second_crash/standby/s123_b: want -:None got f:committed-original
FAIL s124 op=rename end=crash_in_flight ckpt=none reuse=yes
after_crash/standby/s124_a: want f:committed-original got -:None
after_second_crash/standby/s124_a: want f:committed-original got -:None
FAIL s125 op=rename end=crash_in_flight ckpt=before reuse=no
after_crash/standby/s125_a: want f:committed-original got -:None
after_crash/standby/s125_b: want -:None got f:committed-original
after_second_crash/standby/s125_a: want f:committed-original got -:None
after_second_crash/standby/s125_b: want -:None got f:committed-original
FAIL s126 op=rename end=crash_in_flight ckpt=before reuse=yes
after_crash/standby/s126_a: want f:committed-original got -:None
after_second_crash/standby/s126_a: want f:committed-original got -:None
FAIL s127 op=rename end=crash_in_flight ckpt=between reuse=no
after_crash/standby/s127_a: want f:committed-original got -:None
after_crash/standby/s127_b: want -:None got f:committed-original
after_second_crash/standby/s127_a: want f:committed-original got -:None
after_second_crash/standby/s127_b: want -:None got f:committed-original
FAIL s128 op=rename end=crash_in_flight ckpt=between reuse=yes
after_crash/standby/s128_a: want f:committed-original got -:None
after_second_crash/standby/s128_a: want f:committed-original got -:None
FAIL s153 op=write end=crash_in_flight ckpt=none reuse=no
after_crash/standby/s153_w: want f:committed-original got
f:NEW-BYTES-original
after_second_crash/standby/s153_w: want f:committed-original got
f:NEW-BYTES-original
FAIL s154 op=write end=crash_in_flight ckpt=before reuse=no
after_crash/standby/s154_w: want f:committed-original got
f:NEW-BYTES-original
after_second_crash/standby/s154_w: want f:committed-original got
f:NEW-BYTES-original
FAIL s155 op=write end=crash_in_flight ckpt=between reuse=no
after_crash/standby/s155_w: want f:committed-original got
f:NEW-BYTES-original
after_second_crash/standby/s155_w: want f:committed-original got
f:NEW-BYTES-original
===== oracle.py =====
"""Scenario generator, model and oracle for the FILEOPS invariant.
Invariant: after any mix of commit, rollback, crash, checkpoint, 2PC and
standby replay, the files FILEOPS manages are exactly those the committed
transactions imply.
python3 -I oracle.py gen <scenarios.json>
python3 -I oracle.py check <scenarios.json> <dump-dir> [<dump-dir> ...]
The model below knows nothing about FILEOPS internals: it computes the
expected state of each scenario's names from the history alone. The runner
(t/910_oracle_runner.pl) executes the scenarios and dumps what is on disk;
`check` compares the two at every stage, on the primary and on the standby.
"""
import itertools
import json
import sys
from collections import defaultdict
OPS = ["create", "mkdir", "link", "rename", "write", "delete"]
ENDS = ["commit", "rollback", "rollback_to_savepoint", "crash_in_flight",
"crash_after_abort_flush", "commit_prepared", "rollback_prepared"]
CKPTS = ["none", "before", "between"]
REUSE = ["no", "yes"]
CRASH_ENDS = {"crash_in_flight", "crash_after_abort_flush"}
COMMITTED_ENDS = {"commit", "commit_prepared"}
SETUP_CONTENT = "committed-original"
WRITE_CONTENT = "NEW-BYTES"
REUSE_CONTENT = "reused-by-another-xact"
def reuse_target(op, committed):
"""The name that is free after the end, for another transaction to take."""
if committed:
return None
return {"create": "f", "mkdir": "d", "link": "l", "rename": "b"}.get(op)
def scenarios():
out = []
for i, (op, end, ckpt, reuse) in enumerate(itertools.product(OPS, ENDS,
CKPTS, REUSE)):
committed = end in COMMITTED_ENDS
if reuse == "yes" and reuse_target(op, committed) is None:
continue # nothing is free to reuse
out.append({"id": f"s{len(out):03d}", "op": op, "end": end, "ckpt":
ckpt,
"reuse": reuse, "reuse_name": reuse_target(op, committed)})
return out
def model(s):
"""Expected state of the scenario's names: {name: (type, content)}.
type is 'f' (file), 'd' (directory) or '-' (absent)."""
committed = s["end"] in COMMITTED_ENDS
n = lambda x: f"{s['id']}_{x}"
st = {}
op = s["op"]
if op == "create":
st[n("f")] = ("f", "") if committed else ("-", None)
elif op == "mkdir":
st[n("d")] = ("d", None) if committed else ("-", None)
elif op == "link":
st[n("src")] = ("f", SETUP_CONTENT)
st[n("l")] = ("f", SETUP_CONTENT) if committed else ("-", None)
elif op == "rename":
st[n("a")] = ("-", None) if committed else ("f", SETUP_CONTENT)
st[n("b")] = ("f", SETUP_CONTENT) if committed else ("-", None)
elif op == "write":
st[n("w")] = ("f", WRITE_CONTENT + SETUP_CONTENT[len(WRITE_CONTENT):]
if committed else SETUP_CONTENT)
elif op == "delete":
st[n("x")] = ("-", None) if committed else ("f", SETUP_CONTENT)
if s["reuse"] == "yes":
name = n(s["reuse_name"])
st[name] = ("d", None) if op == "mkdir" else ("f", REUSE_CONTENT)
return st
def stages_for(s):
"""Stages at which the scenario's state is defined. 'online' is the
primary right after the end; crash scenarios only exist after the crash."""
first = [] if s["end"] in CRASH_ENDS else [("online", "primary")]
return first + [(st, node) for st in ("after_crash", "after_second_crash")
for node in ("primary", "standby")]
def read_dump(path):
"""Lines: <stage> <node> <name> <type> <content-or-dash>"""
d = defaultdict(dict)
for line in open(path):
stage, node, name, typ, content = line.rstrip("\n").split("\t")
d[(stage, node)][name] = (typ, None if content == "\\N" else content)
return d
def check(spec, dumps):
scen = json.load(open(spec))
runs = [read_dump(f"{d}/dump.tsv") for d in dumps]
result = {}
for s in scen:
want = model(s)
verdicts = []
for run in runs:
bad = []
for stage, node in stages_for(s):
got = run.get((stage, node), {})
for name, w in want.items():
g = got.get(name, ("-", None))
if (g[0], g[1] if w[0] == "f" else None) != (w[0], w[1] if
w[0] == "f" else None):
bad.append(f"{stage}/{node}/{name}: want {w[0]}:{w[1]}
got {g[0]}:{g[1]}")
verdicts.append(bad)
result[s["id"]] = (s, verdicts)
stable_fail, flaky, ok = [], [], []
for sid, (s, v) in result.items():
fails = [bool(b) for b in v]
if all(fails):
stable_fail.append((s, v[0]))
elif any(fails):
flaky.append((s, [b for b in v if b][0]))
else:
ok.append(s)
print(f"scenarios {len(result)} runs {len(runs)} pass {len(ok)} "
f"fail {len(stable_fail)} unstable {len(flaky)}")
for label, group in (("FAIL", stable_fail), ("UNSTABLE", flaky)):
for s, bad in group:
print(f"{label} {s['id']} op={s['op']} end={s['end']}
ckpt={s['ckpt']} reuse={s['reuse']}")
for b in bad[:4]:
print(f" {b}")
if __name__ == "__main__":
if sys.argv[1] == "gen":
sc = scenarios()
json.dump(sc, open(sys.argv[2], "w"), indent=1)
print(f"{len(sc)} scenarios")
elif sys.argv[1] == "check":
check(sys.argv[2], sys.argv[3:])
===== 910_oracle_runner.pl =====
# Runner for the FILEOPS invariant oracle (oracle.py). Executes every
# scenario in SCENARIOS (a JSON file from `oracle.py gen`) on a primary with
# a streaming standby, and writes what is on disk for each scenario's names to
# $ENV{DUMP_DIR}/dump.tsv at three stages:
# online the primary right after each non-crash scenario ends
# after_crash primary and standby after one crash of the primary
# after_second_crash both again after a second crash (replays the same WAL)
# The runner does not judge anything; oracle.py compares against its model.
use strict;
use warnings FATAL => 'all';
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use Test::More;
use JSON::PP;
use Time::HiRes qw(usleep);
my $scen = decode_json(slurp_file($ENV{SCENARIOS}));
my $dumpdir = $ENV{DUMP_DIR} or die 'DUMP_DIR';
open(my $dump, '>', "$dumpdir/dump.tsv") or die;
my $SETUP = 'committed-original';
my $NEW = 'NEW-BYTES';
my $REUSE = 'reused-by-another-xact';
my $primary = PostgreSQL::Test::Cluster->new('primary');
$primary->init(allows_streaming => 1);
$primary->append_conf('postgresql.conf', qq{
max_connections = 250
max_prepared_transactions = 100
checkpoint_timeout = 1h
autovacuum = off
});
$primary->start;
$primary->safe_psql('postgres',
'CREATE EXTENSION test_fileops; CREATE EXTENSION injection_points');
$primary->backup('b');
my $standby = PostgreSQL::Test::Cluster->new('standby');
$standby->init_from_backup($primary, 'b', has_streaming => 1);
$standby->start;
sub sql { $primary->safe_psql('postgres', $_[0]) }
sub nm { my ($s, $x) = @_; return "$s->{id}_$x" }
sub mkfile
{
my ($name, $content) = @_;
sql("BEGIN; SELECT test_fileops_create('$name', 384);"
. " SELECT test_fileops_write('$name', 0, '$content');
COMMIT;");
}
sub setup
{
my ($s) = @_;
my %fixture = (link => 'src', rename => 'a', write => 'w', delete =>
'x');
mkfile(nm($s, $fixture{ $s->{op} }), $SETUP) if $fixture{ $s->{op} };
}
sub op_sql
{
my ($s) = @_;
my $o = $s->{op};
return "SELECT test_fileops_create('" . nm($s, 'f') . "', 384)" if $o
eq 'create';
return "SELECT test_fileops_mkdir('" . nm($s, 'd') . "', 448)" if $o eq
'mkdir';
return "SELECT test_fileops_link('" . nm($s, 'src') . "', '" . nm($s,
'l') . "')" if $o eq 'link';
return "SELECT test_fileops_rename('" . nm($s, 'a') . "', '" . nm($s,
'b') . "')" if $o eq 'rename';
return "SELECT test_fileops_write('" . nm($s, 'w') . "', 0, '$NEW')" if
$o eq 'write';
return "SELECT test_fileops_delete('" . nm($s, 'x') . "')" if $o eq
'delete';
die $o;
}
sub reuse
{
my ($s) = @_;
return unless $s->{reuse} eq 'yes';
my $name = nm($s, $s->{reuse_name});
if ($s->{op} eq 'mkdir') { sql("SELECT test_fileops_mkdir('$name',
448)") }
else { mkfile($name, $REUSE) }
}
sub start_xact
{
my ($s) = @_;
sql('CHECKPOINT') if $s->{ckpt} eq 'before';
my $bg = $primary->background_psql('postgres', on_error_stop => 0);
$bg->query_safe('BEGIN');
$bg->query_safe('SAVEPOINT sp') if $s->{end} eq 'rollback_to_savepoint';
$bg->query_safe(op_sql($s));
sql('CHECKPOINT') if $s->{ckpt} eq 'between';
return $bg;
}
sub dump_state
{
my ($stage, $node) = @_;
my $dir = $node->data_dir;
opendir(my $d, $dir) or die;
for my $e (sort grep { /^s\d{3}_/ } readdir($d))
{
my $p = "$dir/$e";
my ($t, $c) = -d $p ? ('d', '\N') : ('f', slurp_file($p));
print $dump join("\t", $stage, $node->name, $e, $t, $c), "\n";
}
closedir($d);
}
my @crash = grep { $_->{end} =~ /^crash_/ } @$scen;
my @plain = grep { $_->{end} !~ /^crash_/ } @$scen;
setup($_) for @$scen;
# Non-crash scenarios, one at a time.
for my $s (@plain)
{
my $bg = start_xact($s);
my $e = $s->{end};
if ($e eq 'commit') { $bg->query_safe('COMMIT') }
elsif ($e eq 'rollback') { $bg->query_safe('ROLLBACK') }
elsif ($e eq 'rollback_to_savepoint') { $bg->query_safe('ROLLBACK TO
sp; COMMIT') }
else
{
$bg->query_safe("PREPARE TRANSACTION 'g$s->{id}'");
sql(($e eq 'commit_prepared' ? 'COMMIT' : 'ROLLBACK') . "
PREPARED 'g$s->{id}'");
}
$bg->quit;
reuse($s);
}
dump_state('online', $primary);
# Crash scenarios, in rounds that each end with one crash. The injection
# point module has 8 wait slots, so at most 6 aborts are paused per round;
# the in-flight losers all go in the first round.
my @after = grep { $_->{end} eq 'crash_after_abort_flush' } @crash;
my @inflight = grep { $_->{end} eq 'crash_in_flight' } @crash;
my @rounds = ([ @inflight, splice(@after, 0, 6) ]);
push @rounds, [ splice(@after, 0, 6) ] while @after;
for my $round (@rounds)
{
my @open = map { [ $_, start_xact($_) ] } @$round;
sql("SELECT injection_points_attach('fileops-abort-after-flush',
'wait')");
my $waiting = 0;
for my $o (@open)
{
next unless $o->[0]{end} eq 'crash_after_abort_flush';
$o->[1]->query_until(qr/go/, "\\echo go\nROLLBACK;\n");
# A deferred delete has no inverse, so its abort does not pause.
$waiting++ unless $o->[0]{op} eq 'delete';
}
$primary->poll_query_until('postgres',
"SELECT count(*) = $waiting FROM pg_stat_activity WHERE
wait_event = 'fileops-abort-after-flush'")
or die 'aborts did not pause';
note "crash round: " . scalar(@open) . " scenarios, paused after abort
flush: $waiting";
$primary->stop('immediate');
eval { $_->[1]->quit } for @open;
$primary->start;
reuse($_) for @$round;
}
$primary->wait_for_catchup($standby);
dump_state('after_crash', $_) for ($primary, $standby);
$primary->stop('immediate');
$primary->start;
sql('SELECT 1');
$primary->wait_for_catchup($standby);
dump_state('after_second_crash', $_) for ($primary, $standby);
close($dump);
ok(1, 'runner finished');
$standby->stop;
$primary->stop;
done_testing();