Hi,
I encountered two problems with the test_checksums/013_rewind test:
1. It takes ~45 seconds to finish on my machine and it seems timing
doesn't depend the machine it runs on. It is always last finishing
tests on my machine and I need to wait ~30 seconds after all other
tests are done. Some numbers:
Local: 013_rewind -> 45s | 027_stream_regress -> 24s
Windows CI: 013_rewind -> 33s | 027_stream_regress -> 150s
2. The "8 - last common checkpoint is a shutdown checkpoint" test is
flaky. This test fails ~1/10 of the runs on my machine without any
external effort. Error message:
[11:31:29.021](0.007s) not ok 8 - last common checkpoint is a shutdown
checkpoint
[11:31:29.022](0.001s) # Failed test 'last common checkpoint is a
shutdown checkpoint'
# at
/home/nbyavuz/Desktop/projects/postgres/src/test/modules/test_checksums/t/013_rewind.pl
line 161.
[11:31:29.022](0.000s) # ''
# doesn't match '(?^:CHECKPOINT_SHUTDOWN)'
I also saw same error on the CI [1].
----------------------------------------
I spent some time fixing these problems with the help of an LLM.
Problem #1:
We wait for the primary's insert LSN in three places, but the primary
might be idle and not have sent it yet:
1.1:
$node_a->backup('backup');
my $node_b = PostgreSQL::Test::Cluster->new('node_b');
$node_b->init_from_backup($node_a, 'backup', has_streaming => 1);
$node_b->start;
$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
test_checksum_state($node_a, 'off');
test_checksum_state($node_b, 'off');
backup() already flushes the LSN, we can wait for the flush LSN here.
1.2
$node_a->safe_psql('postgres', "CHECKPOINT;");
$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
Since there is already a CHECKPOINT, the changes should be flushed. We
can wait for the flush LSN.
1.3
# Start the rewound node as a standby of the new primary. Replay runs
# through the pre-enable WAL stretch and the online enable.
#
# The rewind replaced the configuration files with those of the new
# primary, so put the port back.
...
$node_a->set_standby_mode;
$node_a->start;
$node_b->wait_for_catchup($node_a, 'replay', $node_b->lsn('insert'));
test_checksum_state($node_a, 'on');
We can use pg_switch_wal() function to make sure changes are flushed
and then we can wait for the flush LSN.
These 3 changes reduces test time from ~45s to ~3s on my local environment.
----------------------------------------
Problem #2
($stdout, $stderr) = run_command(
[
'pg_waldump',
'-p' => $node_a->data_dir . '/pg_wal',
'-t' => 1,
'-s' => $shutdown_ckpt,
'-n' => 1,
]);
like($stdout, qr/CHECKPOINT_SHUTDOWN/,
'last common checkpoint is a shutdown checkpoint');
We don't specifiy which WAL file that pg_waldump() will use, then
pg_waldump select first WAL data available in the directory. Then, it
might select a WAL file whose header is not initialized yet (a
preallocated WAL file). So, when pg_waldump tries to get segment_size
from this file it reads 0 and fails. I run pg_waldump on the failed
test artifacts and I get this error; which I think confirms the
problem:
$ pg_waldump -p
testrun/test_checksums/013_rewind/data/t_013_rewind_node_a_data/pgdata/pg_wal/
pg_waldump: error: invalid WAL segment size in WAL file
"000000020000000000000005" (0 bytes)
pg_waldump: detail: The WAL segment size must be a power of two
between 1 MB and 1 GB.
This problem is solved by specifying the WAL file.
----------------------------------------
Two patchs are attached, 0001 for the problem #1 and 0002 for the problem #2.
[1]
https://github.com/postgres/postgres/actions/runs/35096253076/job/104794579050#step:13:499
--
Regards,
Nazir Bilal Yavuz
Microsoft
From db391b7bb4c4bb0d66bbf65a65086b312882c4f2 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <[email protected]>
Date: Tue, 15 Sep 2026 16:28:28 +0300
Subject: [PATCH v1 1/2] Avoid idle WAL waits in the checksum rewind test
Wait for standby replay through the upstream flush LSN rather than its
insert position, which can include WAL not yet available for streaming
on an idle primary. Backup completion and the explicit checkpoint have
already flushed the WAL required by the first two catch-up waits.
After rewind, explicitly switch WAL on the source before starting the
target. This flushes the WAL needed to reach its minimum recovery point,
avoiding a wait for the background writer to log its next snapshot and
flush the full_page_writes change. Use the source flush LSN for the final
replay wait as well.
---
src/test/modules/test_checksums/t/013_rewind.pl | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/src/test/modules/test_checksums/t/013_rewind.pl b/src/test/modules/test_checksums/t/013_rewind.pl
index a791e24317d..9c2dff78f2c 100644
--- a/src/test/modules/test_checksums/t/013_rewind.pl
+++ b/src/test/modules/test_checksums/t/013_rewind.pl
@@ -71,7 +71,8 @@ my $node_b = PostgreSQL::Test::Cluster->new('node_b');
$node_b->init_from_backup($node_a, 'backup', has_streaming => 1);
$node_b->start;
-$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
+# Backup completion has flushed the required WAL.
+$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('flush'));
test_checksum_state($node_a, 'off');
test_checksum_state($node_b, 'off');
@@ -83,7 +84,7 @@ $node_b->safe_psql('postgres',
# in a background session; it will block on the injection point with
# the checkpointer busy until released.
$node_a->safe_psql('postgres', "CHECKPOINT;");
-$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('insert'));
+$node_a->wait_for_catchup($node_b, 'replay', $node_a->lsn('flush'));
my $bg_psql = $node_b->background_psql('postgres', on_error_stop => 0);
$bg_psql->query_until(
@@ -182,9 +183,14 @@ port = @{[$node_a->port]}
primary_conninfo = '$connstr application_name=@{[$node_a->name]}'
]);
$node_a->set_standby_mode;
+
+# Flush WAL through the minimum recovery point chosen by pg_rewind. The
+# full_page_writes change can leave an unflushed record on the idle source,
+# delaying startup until the background writer logs its next snapshot.
+$node_b->safe_psql('postgres', 'SELECT pg_switch_wal();');
$node_a->start;
-$node_b->wait_for_catchup($node_a, 'replay', $node_b->lsn('insert'));
+$node_b->wait_for_catchup($node_a, 'replay', $node_b->lsn('flush'));
test_checksum_state($node_a, 'on');
is($node_a->safe_psql('postgres', "SELECT count(*) FROM t;"),
--
2.47.3
From 249a574624b8722a5b4df73f010dec277d33a082 Mon Sep 17 00:00:00 2001
From: Nazir Bilal Yavuz <[email protected]>
Date: Thu, 17 Sep 2026 13:23:12 +0300
Subject: [PATCH v1 2/2] Fix WAL file selection in checksum rewind test
Without an explicit WAL filename, pg_waldump determines the segment size
from the first WAL-named directory entry. A preallocated segment with an
uninitialized header can make this fail before the checkpoint is read.
Pass the WAL filename recorded in backup_label to avoid depending on
directory enumeration order. Use command_like() so command failures and
stderr are checked rather than appearing only as an empty-output mismatch.
---
src/test/modules/test_checksums/t/013_rewind.pl | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/src/test/modules/test_checksums/t/013_rewind.pl b/src/test/modules/test_checksums/t/013_rewind.pl
index 9c2dff78f2c..183145c1930 100644
--- a/src/test/modules/test_checksums/t/013_rewind.pl
+++ b/src/test/modules/test_checksums/t/013_rewind.pl
@@ -151,15 +151,21 @@ $backup_label =~ /^CHECKPOINT LOCATION: ([0-9A-F\/]+)$/m
or die "checkpoint location missing from backup_label";
is($1, $shutdown_ckpt, 'replay starts at the switchover checkpoint');
-($stdout, $stderr) = run_command(
+# Specify the WAL file so that pg_waldump does not try to determine the
+# segment size from an arbitrary, possibly preallocated, file in pg_wal.
+$backup_label =~ /^START WAL LOCATION: [0-9A-F\/]+ \(file ([0-9A-F]{24})\)$/m
+ or die "WAL file name missing from backup_label";
+my $shutdown_wal = $1;
+
+command_like(
[
'pg_waldump',
'-p' => $node_a->data_dir . '/pg_wal',
- '-t' => 1,
'-s' => $shutdown_ckpt,
'-n' => 1,
- ]);
-like($stdout, qr/CHECKPOINT_SHUTDOWN/,
+ $shutdown_wal,
+ ],
+ qr/CHECKPOINT_SHUTDOWN/,
'last common checkpoint is a shutdown checkpoint');
# pg_rewind keeps the target's own checksum state in the control file it
--
2.47.3
