Hi Markos, Andrey,
I did some benchmarks and it was somewhat a disappointment, but I'll
summarize it here. There are some improvements, but not as much as I
expected. I leave it to you to decide if it is worth merging. If you
have suggestions for other measurements, I'm happy to do them.
Summary
--------------
Run on 2026-11-08 in `eu-north-1`, on a fresh `m6i.xlarge` with a fresh
50 GiB gp3 volume (3,000 IOPS, 125 MiB/s, XFS), the baseline commit
(unpatched) and the patched commit (read-ahead cache), built as release
(`-O3`, no assertions). 99 timed runs and 4 untimed `strace` count runs.
Cold unless stated otherwise.
Workload: `pgbench -i` bulk load, then `kill -9`. The WAL was generated
with the baseline binary and both binaries recover the same pristine
crashed copy.
Sizes: pgbench scales 25, 100 and 300, i.e. 0.30, 1.19 and 3.57 GiB of
WAL to replay.
Setup: one `m6i.xlarge`, `gp3` volume with 3,000 IOPS and 125 MiB/s, XFS,
crash recovery of 3.57 GiB of WAL (pgbench bulk load), 3 repetitions per
setting, cold page cache unless stated.
Different settings of `io_combine_limit` (cold cache)
| Setting | Dev reads | pread64 | Redo s | CPU s |
| ----------------- | --------- | ------- | ------ | ----- |
| Baseline | 59,921 | 468,121 | 53.5 | 5.82 |
| Patched, 16 pages | 58,779 | 29,408 | 53.2 | 5.76 |
| Patched, 32 pages | 44,168 | 14,784 | 50.1 | 5.46 |
| Patched, 64 pages | 44,156 | 7,472 | 49.4 | 5.18 |
| Baseline, ra=256 | 45,067 | n/c | 50.3 | 5.42 |
Baseline = unpatched build, ra=256 = `read_ahead_kb=256` (default 128).
Dev reads = device read I/Os; CPU = startup process user+system. n/c =
not counted (expected to equal the baseline's, since readahead does not
change the calls).
* Device reads fall 25% at 32 pages. Redo time falls 5.8% (32 pages) and
7.1% (64 pages) and startup CPU 5.3% and 10.1%, compared with the
patch at its default of 16 pages.
* The unpatched baseline with read_ahead_kb=256 has 45,067 device reads
(-24.8%), redo -6.0% and startup CPU -6.8%. The patch with 32 pages is
within 2.0% of that on device reads and 0.3% on redo time. So on this
setup (gp3, XFS, cold cache) `read_ahead_kb=256` gives the same read
reduction without a code change.
* The patch still helps CPU at large limits: with 64 pages, startup CPU
is -4.4% compared with the baseline at read_ahead_kb=256 (95% CI of
the difference [-0.31, -0.16] s). This is probably the saved syscalls.
* Device reads stop falling at 256 KiB: 64 pages gives the same 44,156
reads as 32 pages.
* Time until ready is 66.2 s in every setting. Recovery is limited by
the volume's 125 MiB/s throughput.
* Average IOPS (reads + writes per second) was 1,008 to 1,253 in all
settings, well under the volume's 3,000 limit, and neither the
volume's nor the instance's IOPS limit was exceeded. So fewer reads
did not speed recovery up here.
Sorry about the noise,
Mats Kindahl
On 9/14/26 20:44, Andrey Borodin wrote:
Hi Mats, Markos,
Mats, in the original gp3 setup, did you also measure fewer device-level
reads and shorter recovery time with the normal readahead settings?
I'd also be interested in replay performance with WAL and relation pages
already cached. Fewer syscalls could help even when the kernel already
combines disk reads, but does that saving outweigh the extra per-page
memcpy in this implementation?
Looking at v1, I think there is a cache-validity issue with streaming:
+ if (readSource == XLOG_FROM_STREAM)
+ wanted = Min(wanted, readLen);
...
+ wanted = Max(wanted, XLOG_BLCKSZ);
readLen is the valid prefix of the requested page, not the amount of
available WAL from that page onwards. It is at most XLOG_BLCKSZ, so
these bounds also leave streaming reads at one page regardless of
io_combine_limit.
More importantly, when only part of the page has arrived, pread() still
reads the whole page and readAheadLen records the full result. Once
more WAL arrives, XLogPageRead() can be called again for the same page
with a larger reqLen. The new cache then hits and returns the old copy,
including bytes that were not valid when it was filled.
Should the cache track how many bytes were valid at the time of the
read, and refill when reqLen exceeds that prefix? The available WAL
distance from targetPagePtr to flushedUpto could separately bound the
combined read size.
Thanks!
Best regards, Andrey Borodin.
