On Fri, Oct 19, 2012 at 11:23 AM, Arne Jansen <sensi...@gmx.net> wrote:
> On 19.10.2012 11:16, Irek Szczesniak wrote:
>> On Wed, Oct 17, 2012 at 2:29 PM, Arne Jansen <sensi...@gmx.net> wrote:
>>> We have finished a beta version of the feature. A webrev for it
>>> can be found here:
>>> http://cr.illumos.org/~webrev/sensille/fits-send/
>>> It adds a command 'zfs fits-send'. The resulting streams can
>>> currently only be received on btrfs, but more receivers will
>>> follow.
>>> It would be great if anyone interested could give it some testing
>>> and/or review. If there are no objections, I'll send a formal
>>> webrev soon.
>> Why are you trying to reinvent the wheel? AFAIK some tar versions and
>> AT&T AST pax support deltas based on a standard (I'll have to dig out
>> the exact specification, but from looking at it you did double work).
> I haven't done the research myself, but the result was that pax would
> have needed significant extension, but I don't have the details. If
> you dig out a format already in use that supports everything we need
> (like sharing data between files, needed for btrfs reflinks), it should
> be easy to change the format. Stuffing the data into a specific format
> is not an essential part of the work and can be changed with a limited
> amount of work.
> -Arne
>> Irek

tar/pax was the initial format that was chosen for btrfs send/receive
as it looked like the best and most compatible way. In the middle of
development however I realized that we need more then storing whole
and incremental files/dirs in the format. We needed to store
information about moved, renamed, deleted, reflinked and even partial
clones where only some bits of a file are shared with another. This
can for sure all be implemented in pax, but then the next problem is
that in some situations renamed/moved files need multiple entries to
get to the desired result. For example, file a may be renamed to b
while at the same time file b got renamed to a. In such cases we need
3 entries that use a temporary name so that we don't loose one of the
files while receiving. There are much more complex examples where it
gets quite complicated.

Also, it needed support for metadata (mode, size, uid/gid, ...)
changes on already existing files/dirs. Reusing already existent
tar/pax entry types was not possible for this as standard tar would
overwrite the original files with empty files.

I had all that implemented with pax, using a lot of custom pax
entries. A lot...so many that it didn't look like tar/pax anymore. It
actually mutated from a list of file/dir/link entries (which tar/pax
is meant to be) to a list of filesystem instructions (rename, link,
unlink, rmdir, write parts of a file, clone parts of a file, chmod,

My thought was, that this was already a big misuse of tar/pax, so I
decided to implement a simple format for this purpose only. Using pax
gave no advantages anymore. In tar/pax every entry must have a file
name, even the pax header entries need a file name. The problem now
is, that plain tar will treat every unknown entry type as regular file
and blindly overwrite existing ones which may result in data loss. To
prevent this, I always added something to the file name so that
unpacking with tar would not hurt the user. The unavoidable side
effect however is that the result of a plain untar is unusable without
further interpretation, which will be hard because tar by default does
not dump pax headers but instead ignores unknown entries.

Also, using tar/pax as the format for send/receive may give a user the
wrong impression that he can later use his good old standard tar to
restore his backups...this could be fatal for him.

zfs-discuss mailing list

Reply via email to