Ya, I agree that we need some additional data and testing. The iostat
data in itself doesn't suggest to me that the process (dd) is slow but
rather that most of the data is being retrieved elsewhere (ARC). An
fsstat would be useful to correlate with the iostat data.
One thing that also comes to
Posting this back to zfs-discuss.
Roland's test case (below) is a single threaded sequential write
followed by a single threaded sequential read. His bandwidth
goes from horrible (~2MB/sec) to expected (~30MB/sec)
when prefetch is disabled. This is with relatively recent nv bits (nv110).
Roland