On Mon, Dec 12, 2011 at 11:04 PM, Erik Trimble <tr...@netdemons.com> wrote:
> On 12/12/2011 12:23 PM, Richard Elling wrote:
>> On Dec 11, 2011, at 2:59 PM, Mertol Ozyoney wrote:
>>> Not exactly. What is dedup'ed is the stream only, which is infect not
>>> very
>>> efficient. Real dedup aware replication is taking the necessary steps to
>>> avoid sending a block that exists on the other storage system.

As with all dedup-related performance, the efficiency of various
methods of implementing "zfs send -D" will vary widely, depending on
the dedup-ability of the data, and what is being sent.  However,
sending no blocks that already exist on the target system does seem
like a good goal, since it addresses some use cases that intra-stream
dedup does not.

> (1) when constructing the stream, every time a block is read from a fileset
> (or volume), its checksum is sent to the receiving machine. The receiving
> machine then looks up that checksum in its DDT, and sends back a "needed" or
> "not-needed" reply to the sender. While this lookup is being done, the
> sender must hold the original block in RAM, and cannot write it out to the
> to-be-sent-stream.
> you produce a huge amount of small network packet
> traffic, which trashes network throughput

This seems like a valid approach to me.  When constructing the stream,
the sender need not read the actual data, just the checksum in the
indirect block.  So there is nothing that the sender "must hold in
RAM".  There is no need to create small (or synchronous) network
packets, because sender need not wait for the receiver to determine if
it needs the block or not.  There can be multiple asynchronous
communication streams:  one where the sender sends all the checksums
to the receiver; another where the receiver requests blocks that it
does not have from the sender; and another where the sender sends
requested blocks back to the receiver.  Implementing this may not be
trivial, and in some cases it will not improve on the current
implementation.  But in others it would be a considerable improvement.

zfs-discuss mailing list

Reply via email to