On Jan 24, 2012, at 7:52 AM, Jim Klimov wrote:
> 2012-01-24 13:05, Mickaël CANÉVET wrote:
>> Unless I misunderstood something, zfs send of a volume that has
>> compression activated, uncompress it. So if I do a zfs send|zfs receive
>> from a compressed volume to a compressed volume, my data are
>> uncompressed and compressed again. Right ?
>> Is there a more effective way to do it (without decompression and
>> recompression) ?
> While I can not confirm or deny this statement, it was my
> impression as well. Rationale being that the two systems
> might demand different compression (i.e. "lzjb" or "none"
> on the original system and "gzip-9" on the backup one).
> Just like you probably have different VDEV layouts, etc.
> Or perhaps even different encryption or dedup settings.
that "feature" falls out of the implementation.
> Compression, like many other components, lives on the
> layer "under" logical storage (userdata blocks), and
> gets applied to newly written blocks only (i.e. your
> datasets can have a mix of different compression levels
> for different files or even blocks within a file, if
> you switched the methods during dataset lifetime).
> Actually I would not be surprised if zfs-send userdata
> stream is even above the block level (i.e. it would seem
> normal to me if many small userdata blocks of original
> pool might become one big block on the recipient).
> So while some optimizations are possible, I think they
> would violate layering quite much.
data in the ARC is uncompressed. compression/decompression
occurs in the ZIO pipeline layer below the DSL.
> But, for example, it might make sense for zfs-send to
> include the original compression algorithm information
> into the sent stream and send the compressed data (less
> network traffic or intermediate storage requirement,
> to say the least - at zero price of recompression to
> something perhaps more efficient), and if the recipient
> dataset's algorithm differs - unpack and recompress it
> on the receiving side.
> If that's not done already :)
the compression parameter value is sent, but as you mentioned
above, blocks in a snapshot can be compressed with different
algorithms, so you only actually get the last setting at time of
> So far my over-the-net zfs sends are piped into gzip
> or pigz, ssh and gunzip, and that often speeds up the
> overall transfer. Probably can be done with less overhead
> by "ssh -C" for implementations that have it.
the UNIX philosophy is in play here :-) Sending the data uncompressed
to stdout allows you to pipe it into various transport or transform programs.
ZFS Performance and Training
zfs-discuss mailing list