On 10/06/2014 06:33 PM, Marco Nenciarini wrote:
Il 03/10/14 22:47, Robert Haas ha scritto:
2. Take a differential backup. In the backup label file, note the LSN
of the fullback to which the differential backup is relative, and the
newest LSN guaranteed to be present in the differential backup. The
actual backup can consist of a series of 20-byte buffer tags, those
being the exact set of blocks newer than the base-backup's
latest-guaranteed-to-be-present LSN. Each buffer tag is followed by
an 8kB block of data. If a relfilenode is truncated or removed, you
need some way to indicate that in the backup; e.g. include a buffertag
with forknum = -(forknum + 1) and blocknum = the new number of blocks,
or InvalidBlockNumber if removed entirely.
To have a working backup you need to ship each block which is newer than
latest-guaranteed-to-be-present in full backup and not newer than
latest-guaranteed-to-be-present in the current backup. Also, as a
further optimization, you can think about not sending the empty space in
the middle of each page.
My main concern here is about how postgres can remember that a
relfilenode has been deleted, in order to send the appropriate "deletion
tag".
IMHO the easiest way is to send the full list of files along the backup
and let to the client the task to delete unneeded files. The backup
profile has this purpose.
Right, but the server doesn't need to send a separate backup profile
file for that. Rather, anything that the server *didn't* send, should be
deleted.
I think the missing piece in this puzzle is that even for unmodified
blocks, the server should send a note saying the blocks were present,
but not modified. So for each file present in the server, the server
sends a block stream. For each block, it sends either the full block
contents, if it was modified, or a simple indicator that it was not
modified.
There's a downside to this, though. The client has to read the whole
stream, before it knows which files were present. So when applying a
block stream directly over an old backup, the client cannot delete files
until it has applied all the other changes. That needs more needs more
disk space. With a separate profile file that's sent *before* the rest
of the backup, you could delete the obsolete files first. But that's not
a very big deal. I would suggest that you leave out the profile file in
the first version, and add it as an optimization later, if needed.
Moreover, I do not like the idea of using only a stream of block as the
actual differential backup, for the following reasons:
* AFAIK, with the current infrastructure, you cannot do a backup with a
block stream only. To have a valid backup you need many files for which
the concept of LSN doesn't apply.
Those should be sent in whole. At least in the first version. The
non-relation files are small compared to relation files, so it's not too
bad to just include them in full.
3. Apply a differential backup to a full backup to create an updated
full backup. This is just a matter of scanning the full backup and
the differential backup and applying the changes in the differential
backup to the full backup.
You might want combinations of these, like something that does 2+3 as
a single operation, for efficiency, or a way to copy a full backup and
apply a differential backup to it as you go. But that's it, right?
What else do you need?
Nothing else. Once we agree on definition of involved files and
protocols formats, only the actual coding remains.
BTW, regarding the protocol, I have an idea. Rather than invent a whole
new file format to represent the modified blocks, can we reuse some
existing binary diff file format? For example, the VCDIFF format (RFC
3284). For each unmodified block, the server would send a vcdiff COPY
instruction, to "copy" the block from the old backup, and for a modified
block, the server would send an ADD instruction, with the new block
contents. The VCDIFF file format is quite flexible, but we would only
use a small subset of it. I believe that subset would be just as easy to
generate in the backend as a custom file format, but you could then use
an external tool (xdelta3, open-vcdiff) to apply the diff manually, in
case of emergency. In essence, the server would send a tar stream as
usual, but for each relation file, it would send a VCDIFF file with name
"<relfilenode>.vcdiff" instead.
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
