You can do a bit better with InfiniBand and NFS/RDMA. We are getting
over 1 GB/s across the IB wire writing to the OpenSolaris NFS server and
ZFS when the data is going into RAM. It's the rotating rust that slows
things down.
Carl
----------------------------
Nicholas Lee wrote:
On Wed, Apr 22, 2009 at 5:31 AM, Bob Friesenhahn
<[email protected] <mailto:[email protected]>> wrote:
One obvious wrong assumption is the assumption that the switch is a
"store and forward" device. Modern ethernet switches only consume
enough data to validate the ethernet packet header before they start
sending data on to the destination. This results in delays in the
low microseconds. Unless the outbound switch port is overloaded,
you should ignore the switch entirely. Normal switched ethernet is
full-duplex. There is no need to divide by two for the switch.
You are right, the switch latency for 1500 byte packet is around 14.5us
Modern NFS uses TCP so the Naggle algorithm could become significant
if the writes are very small (not applicable for your large packet).
I have measured NFS write rates of about 90MB/second using 32KB
blocks and using a lowly 4-year old SPARC workstation with Solaris
10 as a client. The individual block writes might not have been
synchronous though since NFSv4 supports a 'commit' feature.
If we turn this around - with 32KB NFS blocks in order to get 90MB you
need about 2900 iops or .35 ms path latency. Network latency (*) on 32KB
is at least .28ms. So .07 ms is left for the sync to disk. I doubt this
is possible with rotational disk - even with a RAID0 array. According to
the numbers I have I'd say it is possible with a strip of eight X25-M
SSD drives or maybe 4 STEC Zeus SSD drives.
Was this number with nfs+zfs or ufs? I thought zfs was more strict with
NFS and did not allow the commit to memory NFS3+ version of sync NFS.
(*) Corrected at 32/125 +.015 = .27 ms. This ignores any overhead, so
lets say .28ms.
Nicholas
------------------------------------------------------------------------
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
_______________________________________________
storage-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/storage-discuss
