Hi list,
I have 2 servers which are directly connected via ixgbe based nics, both
running OpenSolaris 2009.06.
The actual network connection seems fine, iperf reports ~6.3 Gbits/sec
in terms of throughput and nicstat seems to agree that the nics are ~63%
utilized.
Iperf :
henrik at opensolaris:~# ./iperf-2.0.4/src/iperf -c 10.10.10.2 -N -t 40
------------------------------------------------------------
Client connecting to 10.10.10.2, TCP port 5001
TCP window size: 391 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.3 port 56583 connected with 10.10.10.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-40.0 sec 29.3 GBytes 6.29 Gbits/sec
Nicstat :
henrik at naz01:/tmpfs# /export/home/henrik/nicstat -i ixgbe0 2
Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util Sat
21:13:02 ixgbe0 776175 1222.1 96592.9 18961.7 8228.4 66.00 63.7 83018.3
21:13:04 ixgbe0 773081 1217.2 96221.2 18885.3 8227.2 66.00 63.4 82717.5
To measure the NFS throughput over this link I have created a tmpfs
filesystem on the server to avoid the synchronous writes issue as much
as possible.
Client :
henrik at opensolaris:~# mount | grep /nfs
/nfs on 10.10.10.2:/tmpfs
remote/read/write/setuid/devices/forcedirectio/xattr/dev=4dc0007 on Wed Aug 5
20:06:25 2009
Server :
henrik at naz01:/tmpfs# share | grep tmpfs
- /tmpfs sec=sys,root=10.10.10.3 ""
henrik at naz01:/tmpfs# mount | grep tmpfs
/tmpfs on swap read/write/setuid/devices/xattr/dev=4b80006 on Wed Aug 5
21:59:31 2009
I have set the 'forcedirectio' option on the client mount to ensure that
the clients cache gets circumvented.
Using the randomwrite microbenchmark in filebench ($filesize set to 1gb)
I get :
Local on tmpfs :
IO Summary: 5013937 ops, 82738.5 ops/s, (0/82738 r/w) 646.4mb/s, 71us
cpu/op, 0.0ms latency
Tmpfs over NFS :
IO Summary: 383488 ops, 6328.2 ops/s, (0/6328 r/w) 49.4mb/s, 65us cpu/op,
0.2ms latency
These are 2 fully populated 4 socket machines - why the extremely low
transfer speed ?
--
Med venlig hilsen / Best Regards
Henrik Johansen
henrik at scannet.dk
Tlf. 75 53 35 00
ScanNet Group
A/S ScanNet
