On 2026-11-17 09:49 +0200, Jakub Wartak wrote:
On Mon, Nov 16, 2026 at 1:35 PM Markos Fountoulakis
<[email protected]> wrote:

Hello hackers,

This patch improves full-file base backup read performance. The idea is that 
since base backup reads use sequential offsets we can benefit by OS-level 
prefetching. On Linux, the POSIX_FADV_SEQUENTIAL hint increases the FS-level 
readahead window and can improve throughput for read-bound backups.

We do not apply the logic to incremental backups case, since the sequential 
advice may be detrimental to performance due to the potentially random read 
access pattern.

I have benchmarked this patch on an Amazon EC2 instance of type c5.4xlarge with 
a gp3 EBS volume with performance limits of 512MB/s and 4000 IOPS. The OS is 
Ubuntu 24.04 and the filesystem is ext4. The default readahead of the block 
device is 128KB at the OS level.

I built the dataset with pgbench with a scale factor of 100 and performed some 
additional simple updates for 100 seconds which resulted in a PGDATA directory 
size of 2591MiB:
===========================================
/usr/local/pgsql/bin/pgbench -s 100 -i test
/usr/local/pgsql/bin/pgbench --progress=1 -T 100 -j 64 -c 64 
--skip-some-updates test
===========================================
At that point I stopped the postgres server and backed up the dataset (simple 
file copy) for reproducibility.

Before every pg_basebackup run I empty the linux page cache (sync ; echo 3 > 
/proc/sys/vm/drop_caches), start postgres server and time the backup while 
monitoring block device statistics via iostat. WAL path was disabled.


** Vanilla postgres run behaves as follows:
===========================================
$ time /usr/local/pgsql/bin/pg_basebackup --target=blackhole --wal-method=none 
--progress  --checkpoint=fast

real    0m8.029s
===========================================
The effective throughput from statistics divided by time is 197MiB/s.

  From block device statistics:
* maximum request size - 128KB
* maximum throughput - 220MiB/s
* average queue depth - 1.8
* average response time - 1.2 ms


** fadvise sequential enabled behaves as follows:
===========================================
$ time /usr/local/pgsql/bin/pg_basebackup --target=blackhole --wal-method=none 
--progress  --checkpoint=fast

real    0m5.813s
===========================================
The effective throughput from statistics divided by time is 272MiB/s.

  From block device statistics:
* maximum request size - 256KB
* maximum throughput - 300MiB/s
* average queue depth - 1.8
* average response time - 1.5 ms

The optimization seems to be working as expected, the maximum I/O request size 
is doubled and the backup is 38% faster. Backup performance is not doubled 
though, since there are other bottlenecks in play, e.g. many small files that 
don't allow for large prefetching and potential CPU serialization in the backup 
code.

POSIX_FADV_SEQUENTIAL exactly doubles the block device readahead /proc/ entry 
in linux, which is 128KB by default. ext4 uses this configuration entry and the 
observed behaviour matches expectations since maximum READ request size is 
256KB during the backup. I/O queue depth remains the same, so nothing else 
changes in the I/O pattern. Important to note that EBS volumes have a maximum 
I/O size of 256KB so any larger requests would split into smaller ones and 
increase the queue depth as a side-effect.


I did an additional experiment by modifying global filesystem readahead from 
128KB to 4MB as such:
===========================================
echo 4096 > /sys/block/nvme0n1/queue/read_ahead_kb
===========================================

** Vanilla postgres backup with this change performed as follows:
===========================================
postgres@ip-172-31-26-163:~$ time /usr/local/pgsql/bin/pg_basebackup 
--target=blackhole --wal-method=none --progress --checkpoint=fast

real    0m2.891s
===========================================
The effective throughput from statistics divided by time is 548MiB/s.

  From block device statistics:
* maximum request size - 256KB
* maximum throughput - 837MiB/s
* maximum queue depth - 18
* maximum response time - 5.5 ms

Increasing the readahead globally to 4MB makes the backup 178% faster. We 
notice that the EBS throughput limits were temporarily exceeded thanks to burst 
credits that EBS has. It is clear that EBS latency demands higher I/O 
concurrency to fully saturate the underlying disk performance capacity. 
However, it is unclear how other (non-backup) workloads may suffer from such an 
increase of readahead globally. This just shows the potential maximum 
performance we can achieve during a simple backup.

In my opinion, this improvement is a net gain and not a trade-off. Doubling the 
OS-level readeahead only for those sequential accesses halves the number of I/O 
requests and doubles the I/O size in the kernel. This means less I/O 
interference and more efficient kernel block I/O, potentially increasing 
performance for all concurrent I/O workloads and not just for the postgres 
backup running.

Attached is the patch written by Matt Blewitt.

Hi Markos & Matt,

please take a look on [1] thread, one of those patches there is doing the
exact same thing (there's plenty of things there, but one of them is
"pg_basebackup-issue-posix_fadvise-for-more.patch"), but we've also added
support for I/O prefetching for the incremental path too in additional
to just issuing POSIX_FADV_SEQUENTIAL). It would be great if you could
help there too.

-J.

[1] - 
https://www.postgresql.org/message-id/flat/CAKZiRmwwW-hDc3B6ERJB%2BpaX7RNSBcQLheq1KdsTf42cGuRvuA%40mail.gmail.com

Hi Jakub,

That thread([1]) is very interesting, I will definitely review it. Regarding 
the patch in this thread, I still think it's worth merging independently since 
it is low hanging fruit, it is virtually 2 lines of code with measurable 
performance increase, and could also be backpatched trivially into many older 
PG versions with the same benefit.


--
Markos Fountoulakis
PlanetScale



Reply via email to