Ian Collins wrote: > Andrew Gabriel wrote: >> Ian Collins wrote: >>> Andrew Gabriel wrote: >>>> Given the issue described is slow zfs recv over network, I suspect >>>> this is: >>>> >>>> 6729347 Poor zfs receive performance across networks >>>> >>>> This is quite easily worked around by putting a buffering program >>>> between the network and the zfs receive. There is a public domain >>>> "mbuffer" which should work, although I haven't tried it as I wrote >>>> my own. The buffer size you need is about 5 seconds worth of data. >>>> In my case of 7200RPM disks (in a mirror and not striped) and a >>>> gigabit ethernet link, the disks are the limiting factor at around >>>> 57MB/sec sustained i/o, so I used a 250MB buffer to best effect. If >>>> I recall correctly, that speeded up the zfs send/recv across the >>>> network by about 3 times, and it then ran at the disk platter speed. >>> >>> Did this apply to incremental sends as well? I can live with ~20MB/sec >>> for full sends, but ~1MB/sec for incremental sends is a killer. >> It doesn't help the ~1MB/sec periods in incrementals, but it does help >> the fast periods in incrementals. >> > :) > > I don't see the 5 second bursty behaviour described in the bug report. > It's more like 5 second interval gaps in the network traffic while the > data is written to disk.
That is exactly the issue. When the zfs recv data has been written, zfs recv starts reading the network again, but there's only a tiny amount of data buffered in the TCP/IP stack, so it has to wait for the network to heave more data across. In effect, it's a single buffered copy. The addition of a buffer program turns it into a double-buffered (or cyclic buffered) copy, with the disks running flat out continuously, and the network streaming data across continuously at the disk platter speed. What are your theoretical max speeds for network and disk i/o? Taking the smaller of these two, are you seeing the sustained send/recv performance match that (excluding the ~1MB/sec periods which is some other problem)? The effect described in that bug is most obvious when the disk and network speeds are same order of magnitude (as in the example I gave above). Given my disk i/o rate above, if the network is much faster (say, 10GB), then it's going to cope with the bursty nature of the traffic better. If the network is much slower (say, 100MB), then it's going to be running flat out anyway and again you won't notice the bursty reads (a colleague measured only 20% gain in that case, rather than my 200% gain). -- Andrew _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
