Thanks for the reviews!
I had put together a branch with some of feedback (comment clean up,
explicit full run through, etc). But seems like it'll be better to
split it up to allow for some backpatching so scrapped that and
went with the attached.
For the Windows testing I'm stuck with GitHub CI so it's kind of
a pain to cycle through it. Been using the Linux runtimes, total bytes,
and decode counts as a proxy to get some faster feedback.
Latest split out patches are attached. The first one does just the
`--limit=...` where we are only looking for one row. The second one
keeps track of the range of LSN to have smarter start / stop positions.
The third one goes a step further moves some of the bigger pieces out
of the loop so they are not repeated. The comments are cleaned up
as well.
Here's the timings / counts running on Linux:
time stdout decoded per decoded
(mean) via IPC archive scenario total
master 4.46s 139.1 MB 109.8 MB 393.5 MB
patch1 3.70s 57.7 MB 91.1 MB 273.4 MB
patch2 3.33s 37.1 MB 42.6 MB 127.8 MB
patch3 2.51s 19.4 MB 10.8 MB 64.2 MB
Each row includes the patches above it. Patch 1 caps the probe
runs via --limit, patch 2 bounds the scans with marker LSNs,
patch 3 decodes once per check instead of once per scenario.
Running in GitHub CI the Windows runtime twice for those are:
Part 1 - 769s, 247s
Part 2 - 128s, 95s
Part 3 - 121s, 178s
It's a _lot_ of noise in CI, but I think 1 + 2 is already the bulk of the gains.
Look like that's a bit further than Nazir's version as it picks a more
specific LSN for the ranges.
Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/
From ac35321856bfed178d85213737615b6811cb0b83 Mon Sep 17 00:00:00 2001
From: Sehrope Sarkuni <[email protected]>
Date: Tue, 1 Sep 2026 11:42:08 +0000
Subject: [PATCH v3 1/3] Stop pg_waldump TAP test probes after the first record
Several tests only check that a run works or that a message appears.
Without a limit they dump entire segments or the whole WAL range
through IPC::Run, which is very slow on Windows.
---
src/bin/pg_waldump/t/001_basic.pl | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 4fa507cfa20..9d846d2fe42 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -243,8 +243,14 @@ command_fails_like(
[ 'pg_waldump', 'foo', 'bar' ],
qr/error: could not locate WAL file "foo"/,
'start file not found');
-command_like([ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile ],
- qr/./, 'runs with start segment specified');
+command_like(
+ [
+ 'pg_waldump',
+ '--limit' => 1,
+ $node->data_dir . '/pg_wal/' . $start_walfile
+ ],
+ qr/./,
+ 'runs with start segment specified');
command_fails_like(
[ 'pg_waldump', $node->data_dir . '/pg_wal/' . $start_walfile, 'bar' ],
qr/error: could not open file "bar"/,
@@ -252,6 +258,7 @@ command_fails_like(
command_like(
[
'pg_waldump',
+ '--limit' => 1,
$node->data_dir . '/pg_wal/' . $start_walfile,
$node->data_dir . '/pg_wal/' . $end_walfile
],
@@ -260,6 +267,7 @@ command_like(
command_like(
[
'pg_waldump', '--quiet',
+ '--limit' => 1,
'--path', $node->data_dir . '/pg_wal/',
$start_walfile
],
@@ -313,6 +321,7 @@ sub test_pg_waldump_skip_bytes
'--start' => $new_start,
'--end' => $endlsn,
'--path' => $path,
+ '--limit' => 1,
],
'>' => \$stdout,
'2>' => \$stderr;
@@ -425,6 +434,7 @@ for my $scenario (@scenarios)
'--path' => $path,
'--start' => $start_lsn,
'--end' => $end_lsn,
+ '--limit' => 1,
],
qr/./,
'runs with path option and start and end locations');
--
2.43.0
From 4f570db2e6ceb3258f86dd4e6d4283c4775d6cd3 Mon Sep 17 00:00:00 2001
From: Sehrope Sarkuni <[email protected]>
Date: Tue, 1 Sep 2026 11:43:20 +0000
Subject: [PATCH v3 3/3] Decode WAL once per check in the pg_waldump TAP test
Rendering and filtering do not depend on how the input is read. Move
the format check and the filter tests out of the scenario loop and run
them once against the plain directory. Each archive scenario instead
parses the complete range with --quiet, decoding every record without
printing the dump. Coverage of tar and gzip input is unchanged while
each archive is read far fewer times.
---
src/bin/pg_waldump/t/001_basic.pl | 115 ++++++++++++++++++------------
1 file changed, 69 insertions(+), 46 deletions(-)
diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 987bc015369..f677f25be8b 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -393,6 +393,52 @@ sub generate_archive
chdir($cwd) || die "chdir: $!";
}
+# Decode the full range and check the output format once. Rendering
+# does not depend on how the input is read.
+{
+ my @lines = test_pg_waldump($node->data_dir, $start_lsn, $end_lsn);
+ is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
+}
+
+# The filter tests likewise run once. With no limit they check every
+# matching record up to test_end_lsn.
+{
+ my $path = $node->data_dir;
+ my @lines;
+
+ @lines = test_pg_waldump($path, $start_lsn, $test_end_lsn, '--fullpage');
+ is(grep(!/^rmgr:.*\bFPW\b/, @lines), 0, 'all output lines are FPW');
+
+ @lines = test_pg_waldump($path, $start_lsn, $test_end_lsn, '--stats');
+ like($lines[0], qr/WAL statistics/, "statistics on stdout");
+ is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
+
+ @lines =
+ test_pg_waldump($path, $start_lsn, $test_end_lsn, '--stats=record');
+ like($lines[0], qr/WAL statistics/, "statistics on stdout");
+ is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
+
+ @lines =
+ test_pg_waldump($path, $start_lsn, $test_end_lsn, '--rmgr' => 'Btree');
+ is(grep(!/^rmgr: Btree/, @lines), 0, 'only Btree lines');
+
+ # These three can only match records before rel_test_end_lsn.
+ @lines = test_pg_waldump($path, $start_lsn, $rel_test_end_lsn,
+ '--fork' => 'init');
+ is(grep(!/fork init/, @lines), 0, 'only init fork lines');
+
+ @lines = test_pg_waldump($path, $start_lsn, $rel_test_end_lsn,
+ '--relation' => "$default_ts_oid/$postgres_db_oid/$rel_t1_oid");
+ is(grep(!/rel $default_ts_oid\/$postgres_db_oid\/$rel_t1_oid/, @lines),
+ 0, 'only lines for selected relation');
+
+ @lines = test_pg_waldump(
+ $path, $start_lsn, $rel_test_end_lsn,
+ '--relation' => "$default_ts_oid/$postgres_db_oid/$rel_i1a_oid",
+ '--block' => 1);
+ is(grep(!/\bblk 1\b/, @lines), 0, 'only lines for selected block');
+}
+
my $tmp_dir = PostgreSQL::Test::Utils::tempdir_short();
my @scenarios = (
@@ -422,11 +468,11 @@ for my $scenario (@scenarios)
SKIP:
{
- skip "tar command is not available", 56
+ skip "tar command is not available", 25
if (!defined $tar || $tar eq '') && $scenario->{'is_archive'};
skip
"$scenario->{'compression_method'} compression not supported by this build",
- 56
+ 25
if !$scenario->{'enabled'} && $scenario->{'is_archive'};
# create pg_wal archive
@@ -452,7 +498,26 @@ for my $scenario (@scenarios)
],
qr/./,
'runs with path option and start and end locations');
- # The two tests below only need the end of the WAL. Starting at
+
+ # The full range parses without error from each archive. With
+ # --quiet every record is decoded but none are printed. The
+ # plain directory was already fully decoded above.
+ if ($scenario->{'is_archive'})
+ {
+ command_like(
+ [
+ 'pg_waldump', '--quiet',
+ '--path' => $path,
+ '--start' => $start_lsn,
+ '--end' => $end_lsn,
+ ],
+ qr/^$/,
+ 'full WAL range parses successfully');
+ }
+
+ test_pg_waldump_skip_bytes($path, $start_lsn, $end_lsn);
+
+ # The tests below only need the end of the WAL. Starting at
# contrecord_lsn skips the bulk of the filler.
command_fails_like(
[
@@ -472,12 +537,7 @@ for my $scenario (@scenarios)
qr/error: error in WAL record at/,
'errors are shown with --quiet');
- test_pg_waldump_skip_bytes($path, $start_lsn, $end_lsn);
-
- my @lines = test_pg_waldump($path, $start_lsn, $end_lsn);
- is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
-
- @lines = test_pg_waldump($path, $contrecord_lsn, $end_lsn);
+ my @lines = test_pg_waldump($path, $contrecord_lsn, $end_lsn);
is(grep(!/^rmgr: \w/, @lines), 0, 'all output lines are rmgr lines');
test_pg_waldump_skip_bytes($path, $contrecord_lsn, $end_lsn);
@@ -485,43 +545,6 @@ for my $scenario (@scenarios)
@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--limit' => 6);
is(@lines, 6, 'limit option observed');
- # The filter tests read up to test_end_lsn, checking every
- # matching record while skipping the filler.
- @lines =
- test_pg_waldump($path, $start_lsn, $test_end_lsn, '--fullpage');
- is(grep(!/^rmgr:.*\bFPW\b/, @lines), 0, 'all output lines are FPW');
-
- @lines = test_pg_waldump($path, $start_lsn, $test_end_lsn, '--stats');
- like($lines[0], qr/WAL statistics/, "statistics on stdout");
- is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
-
- @lines =
- test_pg_waldump($path, $start_lsn, $test_end_lsn, '--stats=record');
- like($lines[0], qr/WAL statistics/, "statistics on stdout");
- is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
-
- @lines = test_pg_waldump($path, $start_lsn, $test_end_lsn,
- '--rmgr' => 'Btree');
- is(grep(!/^rmgr: Btree/, @lines), 0, 'only Btree lines');
-
- # These three can only match records before rel_test_end_lsn.
- @lines = test_pg_waldump($path, $start_lsn, $rel_test_end_lsn,
- '--fork' => 'init');
- is(grep(!/fork init/, @lines), 0, 'only init fork lines');
-
- @lines = test_pg_waldump($path, $start_lsn, $rel_test_end_lsn,
- '--relation' => "$default_ts_oid/$postgres_db_oid/$rel_t1_oid");
- is( grep(!/rel $default_ts_oid\/$postgres_db_oid\/$rel_t1_oid/,
- @lines),
- 0,
- 'only lines for selected relation');
-
- @lines = test_pg_waldump(
- $path, $start_lsn, $rel_test_end_lsn,
- '--relation' => "$default_ts_oid/$postgres_db_oid/$rel_i1a_oid",
- '--block' => 1);
- is(grep(!/\bblk 1\b/, @lines), 0, 'only lines for selected block');
-
# Cleanup.
unlink $path if $scenario->{'is_archive'};
}
--
2.43.0
From 83c958170b6f680e969be665c4f6fc410de5c63d Mon Sep 17 00:00:00 2001
From: Sehrope Sarkuni <[email protected]>
Date: Tue, 1 Sep 2026 11:43:09 +0000
Subject: [PATCH v3 2/3] Bound pg_waldump TAP test decodes to the records under
test
The filter tests cannot match anything past the workload generated for
them, and three of them cannot match anything past the t1 and t2
records. Capture an LSN at each boundary and stop reading there. The
fall-off-the-end checks start at the contrecord LSN because they only
exercise the WAL tail.
---
src/bin/pg_waldump/t/001_basic.pl | 42 +++++++++++++++++++++++--------
1 file changed, 31 insertions(+), 11 deletions(-)
diff --git a/src/bin/pg_waldump/t/001_basic.pl b/src/bin/pg_waldump/t/001_basic.pl
index 9d846d2fe42..987bc015369 100644
--- a/src/bin/pg_waldump/t/001_basic.pl
+++ b/src/bin/pg_waldump/t/001_basic.pl
@@ -140,7 +140,16 @@ ROLLBACK;
CREATE UNLOGGED TABLE t2 (x int);
CREATE INDEX i2 ON t2 USING btree (x);
INSERT INTO t2 SELECT generate_series(1, 10);
+});
+
+# Everything the --relation, --block and --fork tests can match lies
+# before this point. The TRUNCATE changed the relfilenodes of t1 and
+# i1a, and only t2 and i2 have init forks.
+my $rel_test_end_lsn =
+ $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()');
+$node->safe_psql(
+ 'postgres', q{
-- gin
CREATE TABLE gin_idx_tbl (id bigserial PRIMARY KEY, data jsonb);
CREATE INDEX gin_idx ON gin_idx_tbl USING gin (data);
@@ -185,6 +194,11 @@ CREATE TABLESPACE ts1 LOCATION '$tblspc_path';
DROP TABLESPACE ts1;
});
+# The option and filter tests read up to here. The WAL beyond this
+# point only sets up the contrecord case.
+my $test_end_lsn =
+ $node->safe_psql('postgres', 'SELECT pg_current_wal_insert_lsn()');
+
# Test: Decode a continuation record (contrecord) that spans multiple WAL
# segments.
#
@@ -438,11 +452,13 @@ for my $scenario (@scenarios)
],
qr/./,
'runs with path option and start and end locations');
+ # The two tests below only need the end of the WAL. Starting at
+ # contrecord_lsn skips the bulk of the filler.
command_fails_like(
[
'pg_waldump',
'--path' => $path,
- '--start' => $start_lsn,
+ '--start' => $contrecord_lsn,
],
qr/error: error in WAL record at/,
'falling off the end of the WAL results in an error');
@@ -451,7 +467,7 @@ for my $scenario (@scenarios)
[
'pg_waldump', '--quiet',
'--path' => $path,
- '--start' => $start_lsn
+ '--start' => $contrecord_lsn
],
qr/error: error in WAL record at/,
'errors are shown with --quiet');
@@ -469,27 +485,31 @@ for my $scenario (@scenarios)
@lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--limit' => 6);
is(@lines, 6, 'limit option observed');
- @lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--fullpage');
+ # The filter tests read up to test_end_lsn, checking every
+ # matching record while skipping the filler.
+ @lines =
+ test_pg_waldump($path, $start_lsn, $test_end_lsn, '--fullpage');
is(grep(!/^rmgr:.*\bFPW\b/, @lines), 0, 'all output lines are FPW');
- @lines = test_pg_waldump($path, $start_lsn, $end_lsn, '--stats');
+ @lines = test_pg_waldump($path, $start_lsn, $test_end_lsn, '--stats');
like($lines[0], qr/WAL statistics/, "statistics on stdout");
is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
@lines =
- test_pg_waldump($path, $start_lsn, $end_lsn, '--stats=record');
+ test_pg_waldump($path, $start_lsn, $test_end_lsn, '--stats=record');
like($lines[0], qr/WAL statistics/, "statistics on stdout");
is(grep(/^rmgr:/, @lines), 0, 'no rmgr lines output');
- @lines =
- test_pg_waldump($path, $start_lsn, $end_lsn, '--rmgr' => 'Btree');
+ @lines = test_pg_waldump($path, $start_lsn, $test_end_lsn,
+ '--rmgr' => 'Btree');
is(grep(!/^rmgr: Btree/, @lines), 0, 'only Btree lines');
- @lines =
- test_pg_waldump($path, $start_lsn, $end_lsn, '--fork' => 'init');
+ # These three can only match records before rel_test_end_lsn.
+ @lines = test_pg_waldump($path, $start_lsn, $rel_test_end_lsn,
+ '--fork' => 'init');
is(grep(!/fork init/, @lines), 0, 'only init fork lines');
- @lines = test_pg_waldump($path, $start_lsn, $end_lsn,
+ @lines = test_pg_waldump($path, $start_lsn, $rel_test_end_lsn,
'--relation' => "$default_ts_oid/$postgres_db_oid/$rel_t1_oid");
is( grep(!/rel $default_ts_oid\/$postgres_db_oid\/$rel_t1_oid/,
@lines),
@@ -497,7 +517,7 @@ for my $scenario (@scenarios)
'only lines for selected relation');
@lines = test_pg_waldump(
- $path, $start_lsn, $end_lsn,
+ $path, $start_lsn, $rel_test_end_lsn,
'--relation' => "$default_ts_oid/$postgres_db_oid/$rel_i1a_oid",
'--block' => 1);
is(grep(!/\bblk 1\b/, @lines), 0, 'only lines for selected block');
--
2.43.0
