Hello all, With the current implementation of pg_combinebackup, we have a few copy methods: --clone, --copy and --copy-file-range. By using either of them, it implicates an actual file copy in the file system, i.e. among other things, disk usage.
While discussing with some people, e.g. Robert Haas and Martín Marqués, about possible ways to improve pg_combinebackup performance and/or reduce disk usage taken by the synthetic backup, there was a thought of using hard links. I'm submitting a patch in this thread which introduces the -k/--link option to pg_combinebackup, making it similar to the options exposed by pg_upgrade. The patch reuses the same code flow that already exists in pg_combinebackup, and whenever the tool judges a file should be copied from either of the input backups to the synthetic backup, if the link mode was chosen, a hard link is created instead of performing a copy. Depending on the pattern of modification of PGDATA files between backups (based on the workload), the user might face improved performance of pg_combinebackup as well as lower disk usage by the synthetic backup. That enhancement comes at a cost: the input backups may be invalidated if the user modified the shared files from the synthetic backup, or if the user starts the cluster from the synthetic backup. With that in mind, a warning has been added to pg_combinebackup, as well as to the docs, recommending that the user moves the synthetic backup to another file system or machine before making use of it. I've run the existing test suite with --link option and it worked fine, except for an issue when running pg_verifybackup in t/003_timeline.pl. The issue was expected given the description shared above. I modified the tests to behave slightly differently depending on the copy method selected by the runner. Besides that, you can find below the outcomes of manual testing: * Create some tables: postgres=# CREATE TABLE test_1 AS SELECT generate_series(1, 1000000); SELECT 1000000 postgres=# CREATE TABLE test_2 AS SELECT generate_series(1, 1000000); SELECT 1000000 postgres=# CREATE TABLE test_3 AS SELECT generate_series(1, 1000000); SELECT 1000000 * Check their OID: postgres=# SELECT oid, relname FROM pg_class WHERE relname LIKE 'test_%'; oid | relname -------+--------- 16388 | test_1 16391 | test_2 16394 | test_3 (3 rows) * Enable the WAL summarizer: postgres=# ALTER SYSTEM SET summarize_wal TO on; ALTER SYSTEM postgres=# SELECT pg_reload_conf(); pg_reload_conf ---------------- t (1 row) * Take a full backup: $ pg_basebackup -d 'dbname=postgres' -D ~/monday -c fast -Xn NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup * Perform an update that will touch the whole file: postgres=# UPDATE test_2 SET generate_series = generate_series + 1; UPDATE 1000000 * Perform an update that will touch a single page: postgres=# UPDATE test_3 SET generate_series = generate_series + 1 WHERE generate_series = 1; UPDATE 1 * Take an incremental backup: $ pg_basebackup -d 'dbname=postgres' -D ~/tuesday -c fast -Xn --incremental ~/monday/backup_manifest NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup * Check a dry run of pg_combinebackup, focusing on the OIDs of the tables: $ pg_combinebackup -d -n --link -o ~/tuesday_full ~/monday ~/tuesday/ 2>&1 | grep -P '16388|16391|16394' pg_combinebackup: would copy "/home/vagrant/monday/base/5/16388" to "/home/vagrant/tuesday_full/base/5/16388" using strategy link pg_combinebackup: would copy "/home/vagrant/tuesday//base/5/16391" to "/home/vagrant/tuesday_full/base/5/16391" using strategy link pg_combinebackup: would reconstruct "/home/vagrant/tuesday_full/base/5/16394" (4480 blocks, checksum CRC32C) pg_combinebackup: reconstruction plan: 0:/home/vagrant/tuesday//base/5/INCREMENTAL.16394@8192 1-4423:/home/vagrant/monday/base/5/16394@36233216 4424:/home/vagrant/tuesday//base/5/INCREMENTAL.16394@16384 4425-4479:/home/vagrant/monday/base/5/16394@36691968 pg_combinebackup: would have read 4478 blocks from "/home/vagrant/monday/base/5/16394" pg_combinebackup: would have read 2 blocks from "/home/vagrant/tuesday//base/5/INCREMENTAL.16394" pg_combinebackup: would copy "/home/vagrant/monday/base/5/16388_vm" to "/home/vagrant/tuesday_full/base/5/16388_vm" using strategy link pg_combinebackup: would copy "/home/vagrant/tuesday//base/5/16388_fsm" to "/home/vagrant/tuesday_full/base/5/16388_fsm" using strategy link pg_combinebackup: would copy "/home/vagrant/tuesday//base/5/16391_vm" to "/home/vagrant/tuesday_full/base/5/16391_vm" using strategy link pg_combinebackup: would copy "/home/vagrant/tuesday//base/5/16391_fsm" to "/home/vagrant/tuesday_full/base/5/16391_fsm" using strategy link pg_combinebackup: would copy "/home/vagrant/monday/base/5/16394_vm" to "/home/vagrant/tuesday_full/base/5/16394_vm" using strategy link pg_combinebackup: would copy "/home/vagrant/tuesday//base/5/16394_fsm" to "/home/vagrant/tuesday_full/base/5/16394_fsm" using strategy link * Create a synthetic backup: $ pg_combinebackup --link -o ~/tuesday_full ~/monday ~/tuesday pg_combinebackup: warning: Modifying files or starting the cluster from the synthetic backup might invalidate one or more of the input backups pg_combinebackup: hint: It is recommended to move the synthetic backup to another file system or machine before performing changes and/or starting the cluster * Check files with link counter set to 1: $ find ~/tuesday_full -type f -links 1 /home/vagrant/tuesday_full/backup_label /home/vagrant/tuesday_full/base/5/1259 /home/vagrant/tuesday_full/base/5/2619 /home/vagrant/tuesday_full/base/5/16394 /home/vagrant/tuesday_full/backup_manifest * Check files with link counter set to 2, focusing on the OIDs of the tables: $ find ~/tuesday_full -type f -links 2 | grep -P '16388|16391|16394' /home/vagrant/tuesday_full/base/5/16388 /home/vagrant/tuesday_full/base/5/16391 /home/vagrant/tuesday_full/base/5/16388_vm /home/vagrant/tuesday_full/base/5/16388_fsm /home/vagrant/tuesday_full/base/5/16391_vm /home/vagrant/tuesday_full/base/5/16391_fsm /home/vagrant/tuesday_full/base/5/16394_vm /home/vagrant/tuesday_full/base/5/16394_fsm (There are many other linked files).
v1-0001-pg_combinebackup-add-support-for-hard-links.patch
Description: Binary data
