On Mon, Dec 12, 2011 at 11:04 PM, Erik Trimble <tr...@netdemons.com> wrote: > On 12/12/2011 12:23 PM, Richard Elling wrote: >> >> On Dec 11, 2011, at 2:59 PM, Mertol Ozyoney wrote: >> >>> Not exactly. What is dedup'ed is the stream only, which is infect not >>> very >>> efficient. Real dedup aware replication is taking the necessary steps to >>> avoid sending a block that exists on the other storage system.
As with all dedup-related performance, the efficiency of various methods of implementing "zfs send -D" will vary widely, depending on the dedup-ability of the data, and what is being sent. However, sending no blocks that already exist on the target system does seem like a good goal, since it addresses some use cases that intra-stream dedup does not. > (1) when constructing the stream, every time a block is read from a fileset > (or volume), its checksum is sent to the receiving machine. The receiving > machine then looks up that checksum in its DDT, and sends back a "needed" or > "not-needed" reply to the sender. While this lookup is being done, the > sender must hold the original block in RAM, and cannot write it out to the > to-be-sent-stream. ... > you produce a huge amount of small network packet > traffic, which trashes network throughput This seems like a valid approach to me. When constructing the stream, the sender need not read the actual data, just the checksum in the indirect block. So there is nothing that the sender "must hold in RAM". There is no need to create small (or synchronous) network packets, because sender need not wait for the receiver to determine if it needs the block or not. There can be multiple asynchronous communication streams: one where the sender sends all the checksums to the receiver; another where the receiver requests blocks that it does not have from the sender; and another where the sender sends requested blocks back to the receiver. Implementing this may not be trivial, and in some cases it will not improve on the current implementation. But in others it would be a considerable improvement. --matt _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
