In my own experiments with my own equivalent of mbuffer, it's well worth giving the receiving side a buffer which is sized to hold the amount of data in a transaction commit, which allows ZFS to be banging out one tx group to disk, whilst the network is bringing the next one across for it. This will be roughly the link speed in bytes/second x 5, plus a bit more for good measure, say 250-300Mbytes for a gigabit link. It seems to be most important when the disks and the network link have similar max theoretical bandwidths (100Mbytes/sec is what you might expect from both gigabit ethernet and reasonable disks), and it becomes less important as the difference in max performance between them increases. Without the buffer, you tend to see the network run flat out for 5 seconds, and then the receiving disks run flat out for 5 seconds, alternating back and forth, whereas with the buffer, both continue streaming at full gigabit speed without a break.

I have not seen any benefit of buffering on the sending side, although I'd still be inclined to include a small one.


Palmer, Trey wrote:
We have found mbuffer to be the fastest solution.   Our rates for large 
transfers on 10GbE are:

280MB/s    mbuffer
220MB/s    rsh
180MB/s    HPN-ssh unencrypted
 60MB/s     standard ssh

The tradeoff mbuffer is a little more complicated to script; rsh is, well, you know; and hpn-ssh requires rebuilding ssh and (probably) maintaining a second copy of it.

Andrew Gabriel
zfs-discuss mailing list

Reply via email to