On Mon, Jul 19, 2010 at 5:20 PM, Richard Jahnel <rich...@ellipseinc.com> wrote: >>I've used mbuffer to transfer hundreds of TB without a problem in mbuffer >>itself. You will get disconnected if the send or receive prematurely ends, >>though. > > > mbuffer itself very specifically ends with a broken pipe error. Very quickly > with s set to 128 or after sometime with s set over 1024. > > My only thought at this point is it maybe something with either the myri10ge > driver or the fact the machines are directly connected with out the use of an > intervening switch. > > Perhaps someone over in networking my have a thought in that direction.
Is the receiving end able to write to disk fast enough ? I have seem those problems in a pool with only two raidz2 vdevs (with 11 disks each) and a 1Gbps connection. ZFS would not be able to write fast enough which would fill all the receiving buffers. To confirm this you can either sniff your traffic and look for ZeroWindow [1] packets or check it with netstat (Rwind should be close to zero). If you transfer with mbuffer and set the buffer to 2GB, you'll see that the sending side is able to transfer at full network speed until the buffer is totally filled. [1] - http://wiki.wireshark.org/TCP%20ZeroWindow -- Giovanni Tirloni gtirl...@sysdroid.com _______________________________________________ networking-discuss mailing list networking-discuss@opensolaris.org
