On 2026-11-17 09:49 +0200, Jakub Wartak wrote:
On Mon, Nov 16, 2026 at 1:35 PM Markos Fountoulakis
<[email protected]> wrote:
Hello hackers,
This patch improves full-file base backup read performance. The idea is that
since base backup reads use sequential offsets we can benefit by OS-level
prefetching. On Linux, the POSIX_FADV_SEQUENTIAL hint increases the FS-level
readahead window and can improve throughput for read-bound backups.
We do not apply the logic to incremental backups case, since the sequential
advice may be detrimental to performance due to the potentially random read
access pattern.
I have benchmarked this patch on an Amazon EC2 instance of type c5.4xlarge with
a gp3 EBS volume with performance limits of 512MB/s and 4000 IOPS. The OS is
Ubuntu 24.04 and the filesystem is ext4. The default readahead of the block
device is 128KB at the OS level.
I built the dataset with pgbench with a scale factor of 100 and performed some
additional simple updates for 100 seconds which resulted in a PGDATA directory
size of 2591MiB:
===========================================
/usr/local/pgsql/bin/pgbench -s 100 -i test
/usr/local/pgsql/bin/pgbench --progress=1 -T 100 -j 64 -c 64
--skip-some-updates test
===========================================
At that point I stopped the postgres server and backed up the dataset (simple
file copy) for reproducibility.
Before every pg_basebackup run I empty the linux page cache (sync ; echo 3 >
/proc/sys/vm/drop_caches), start postgres server and time the backup while
monitoring block device statistics via iostat. WAL path was disabled.
** Vanilla postgres run behaves as follows:
===========================================
$ time /usr/local/pgsql/bin/pg_basebackup --target=blackhole --wal-method=none
--progress --checkpoint=fast
real 0m8.029s
===========================================
The effective throughput from statistics divided by time is 197MiB/s.
From block device statistics:
* maximum request size - 128KB
* maximum throughput - 220MiB/s
* average queue depth - 1.8
* average response time - 1.2 ms
** fadvise sequential enabled behaves as follows:
===========================================
$ time /usr/local/pgsql/bin/pg_basebackup --target=blackhole --wal-method=none
--progress --checkpoint=fast
real 0m5.813s
===========================================
The effective throughput from statistics divided by time is 272MiB/s.
From block device statistics:
* maximum request size - 256KB
* maximum throughput - 300MiB/s
* average queue depth - 1.8
* average response time - 1.5 ms
The optimization seems to be working as expected, the maximum I/O request size
is doubled and the backup is 38% faster. Backup performance is not doubled
though, since there are other bottlenecks in play, e.g. many small files that
don't allow for large prefetching and potential CPU serialization in the backup
code.
POSIX_FADV_SEQUENTIAL exactly doubles the block device readahead /proc/ entry
in linux, which is 128KB by default. ext4 uses this configuration entry and the
observed behaviour matches expectations since maximum READ request size is
256KB during the backup. I/O queue depth remains the same, so nothing else
changes in the I/O pattern. Important to note that EBS volumes have a maximum
I/O size of 256KB so any larger requests would split into smaller ones and
increase the queue depth as a side-effect.
I did an additional experiment by modifying global filesystem readahead from
128KB to 4MB as such:
===========================================
echo 4096 > /sys/block/nvme0n1/queue/read_ahead_kb
===========================================
** Vanilla postgres backup with this change performed as follows:
===========================================
postgres@ip-172-31-26-163:~$ time /usr/local/pgsql/bin/pg_basebackup
--target=blackhole --wal-method=none --progress --checkpoint=fast
real 0m2.891s
===========================================
The effective throughput from statistics divided by time is 548MiB/s.
From block device statistics:
* maximum request size - 256KB
* maximum throughput - 837MiB/s
* maximum queue depth - 18
* maximum response time - 5.5 ms
Increasing the readahead globally to 4MB makes the backup 178% faster. We
notice that the EBS throughput limits were temporarily exceeded thanks to burst
credits that EBS has. It is clear that EBS latency demands higher I/O
concurrency to fully saturate the underlying disk performance capacity.
However, it is unclear how other (non-backup) workloads may suffer from such an
increase of readahead globally. This just shows the potential maximum
performance we can achieve during a simple backup.
In my opinion, this improvement is a net gain and not a trade-off. Doubling the
OS-level readeahead only for those sequential accesses halves the number of I/O
requests and doubles the I/O size in the kernel. This means less I/O
interference and more efficient kernel block I/O, potentially increasing
performance for all concurrent I/O workloads and not just for the postgres
backup running.
Attached is the patch written by Matt Blewitt.
Hi Markos & Matt,
please take a look on [1] thread, one of those patches there is doing the
exact same thing (there's plenty of things there, but one of them is
"pg_basebackup-issue-posix_fadvise-for-more.patch"), but we've also added
support for I/O prefetching for the incremental path too in additional
to just issuing POSIX_FADV_SEQUENTIAL). It would be great if you could
help there too.
-J.
[1] -
https://www.postgresql.org/message-id/flat/CAKZiRmwwW-hDc3B6ERJB%2BpaX7RNSBcQLheq1KdsTf42cGuRvuA%40mail.gmail.com
Hi Jakub,
That thread([1]) is very interesting, I will definitely review it. Regarding
the patch in this thread, I still think it's worth merging independently since
it is low hanging fruit, it is virtually 2 lines of code with measurable
performance increase, and could also be backpatched trivially into many older
PG versions with the same benefit.
--
Markos Fountoulakis
PlanetScale