I'm thinking of implementing an incremental backup tool for
PostgreSQL. The use case for the tool would be taking a backup of huge
database. For that size of database, pg_dump is too slow, even WAL
archive is too slow/ineffective as well. However even in a TB
database, sometimes actual modified blocks are not that big, may be
even several GB. So if we can backup those modified blocks only,
that would be an effective incremental backup method.

For now, my idea is pretty vague.

- Record info about modified blocks. We don't need to remember the
  whole history of a block if the block was modified multiple times.
  We just remember that the block was modified since the last
  incremental backup was taken.

- The info could be obtained by trapping calls to mdwrite() etc. We need
  to be careful to avoid such blocks used in xlogs and temporary
  tables to not waste resource.

- If many blocks were modified in a file, we may be able to condense
  the info as "the whole file was modified" to reduce the amount of

- How to take a consistent incremental backup is an issue. I can't
  think of a clean way other than "locking whole cluster", which is
  obviously unacceptable. Maybe we should give up "hot backup"?

Comments, thoughts are welcome.
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:

Reply via email to