On Mon, Nov 16, 2026 at 1:35 PM Markos Fountoulakis <[email protected]> wrote: > > Hello hackers, > > This patch improves full-file base backup read performance. The idea is that > since base backup reads use sequential offsets we can benefit by OS-level > prefetching. On Linux, the POSIX_FADV_SEQUENTIAL hint increases the FS-level > readahead window and can improve throughput for read-bound backups. > > We do not apply the logic to incremental backups case, since the sequential > advice may be detrimental to performance due to the potentially random read > access pattern. > > I have benchmarked this patch on an Amazon EC2 instance of type c5.4xlarge > with a gp3 EBS volume with performance limits of 512MB/s and 4000 IOPS. The > OS is Ubuntu 24.04 and the filesystem is ext4. The default readahead of the > block device is 128KB at the OS level. > > I built the dataset with pgbench with a scale factor of 100 and performed > some additional simple updates for 100 seconds which resulted in a PGDATA > directory size of 2591MiB: > =========================================== > /usr/local/pgsql/bin/pgbench -s 100 -i test > /usr/local/pgsql/bin/pgbench --progress=1 -T 100 -j 64 -c 64 > --skip-some-updates test > =========================================== > At that point I stopped the postgres server and backed up the dataset (simple > file copy) for reproducibility. > > Before every pg_basebackup run I empty the linux page cache (sync ; echo 3 > > /proc/sys/vm/drop_caches), start postgres server and time the backup while > monitoring block device statistics via iostat. WAL path was disabled. > > > ** Vanilla postgres run behaves as follows: > =========================================== > $ time /usr/local/pgsql/bin/pg_basebackup --target=blackhole > --wal-method=none --progress --checkpoint=fast > > real 0m8.029s > =========================================== > The effective throughput from statistics divided by time is 197MiB/s. > > From block device statistics: > * maximum request size - 128KB > * maximum throughput - 220MiB/s > * average queue depth - 1.8 > * average response time - 1.2 ms > > > ** fadvise sequential enabled behaves as follows: > =========================================== > $ time /usr/local/pgsql/bin/pg_basebackup --target=blackhole > --wal-method=none --progress --checkpoint=fast > > real 0m5.813s > =========================================== > The effective throughput from statistics divided by time is 272MiB/s. > > From block device statistics: > * maximum request size - 256KB > * maximum throughput - 300MiB/s > * average queue depth - 1.8 > * average response time - 1.5 ms > > The optimization seems to be working as expected, the maximum I/O request > size is doubled and the backup is 38% faster. Backup performance is not > doubled though, since there are other bottlenecks in play, e.g. many small > files that don't allow for large prefetching and potential CPU serialization > in the backup code. > > POSIX_FADV_SEQUENTIAL exactly doubles the block device readahead /proc/ entry > in linux, which is 128KB by default. ext4 uses this configuration entry and > the observed behaviour matches expectations since maximum READ request size > is 256KB during the backup. I/O queue depth remains the same, so nothing else > changes in the I/O pattern. Important to note that EBS volumes have a maximum > I/O size of 256KB so any larger requests would split into smaller ones and > increase the queue depth as a side-effect. > > > I did an additional experiment by modifying global filesystem readahead from > 128KB to 4MB as such: > =========================================== > echo 4096 > /sys/block/nvme0n1/queue/read_ahead_kb > =========================================== > > ** Vanilla postgres backup with this change performed as follows: > =========================================== > postgres@ip-172-31-26-163:~$ time /usr/local/pgsql/bin/pg_basebackup > --target=blackhole --wal-method=none --progress --checkpoint=fast > > real 0m2.891s > =========================================== > The effective throughput from statistics divided by time is 548MiB/s. > > From block device statistics: > * maximum request size - 256KB > * maximum throughput - 837MiB/s > * maximum queue depth - 18 > * maximum response time - 5.5 ms > > Increasing the readahead globally to 4MB makes the backup 178% faster. We > notice that the EBS throughput limits were temporarily exceeded thanks to > burst credits that EBS has. It is clear that EBS latency demands higher I/O > concurrency to fully saturate the underlying disk performance capacity. > However, it is unclear how other (non-backup) workloads may suffer from such > an increase of readahead globally. This just shows the potential maximum > performance we can achieve during a simple backup. > > In my opinion, this improvement is a net gain and not a trade-off. Doubling > the OS-level readeahead only for those sequential accesses halves the number > of I/O requests and doubles the I/O size in the kernel. This means less I/O > interference and more efficient kernel block I/O, potentially increasing > performance for all concurrent I/O workloads and not just for the postgres > backup running. > > Attached is the patch written by Matt Blewitt.
Hi Markos & Matt, please take a look on [1] thread, one of those patches there is doing the exact same thing (there's plenty of things there, but one of them is "pg_basebackup-issue-posix_fadvise-for-more.patch"), but we've also added support for I/O prefetching for the incremental path too in additional to just issuing POSIX_FADV_SEQUENTIAL). It would be great if you could help there too. -J. [1] - https://www.postgresql.org/message-id/flat/CAKZiRmwwW-hDc3B6ERJB%2BpaX7RNSBcQLheq1KdsTf42cGuRvuA%40mail.gmail.com
