On Sun, Jan 1, 2012 at 6:13 AM, Magnus Hagander <mag...@hagander.net> wrote: > It also doesn't affect backups taken through pg_basebackup - but I > guess you have good reasons for not being able to use that?
Parallel archiving/de-archiving and segmentation of the backup into pieces and rate limiting are the most clear gaps. I don't know if there are performance implications either, but I do pass all my bytes through unoptimized Python right now -- not exactly a speed demon. The approach I use is: * Scan the directory tree immediately after pg_start_backup, taking notes of existent files and sizes * Split those files into volumes, none of which can exceed 1.5GB. These volumes are all disjoint * When creating the tar file, set the header for a tar member to have as many bytes as recorded in the first pass. If the file has been truncated, pad with zeros (this is also the behavior of GNU Tar). If it grew, only read the number of bytes recorded. * Generate and compress these tar files in parallel * All the while, the rate of reading files is subject to optional rate limiting As important is the fact that each volume can be downloaded and decompressed in a pipeline (no on-disk transformations to de-archive) with a tunable amount of concurrency, as all that tar files do not overlap for any file, and no file needs to span two tar files thanks to Postgres's refusal to deal in files too large for old platforms. -- fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
