On 08/05/09 15:53, Henrik Johansen wrote: > Hi list, > > I have 2 servers which are directly connected via ixgbe based nics, both > running OpenSolaris 2009.06. > > The actual network connection seems fine, iperf reports ~6.3 Gbits/sec > in terms of throughput and nicstat seems to agree that the nics are ~63% > utilized. > Iperf : henrik at opensolaris:~# ./iperf-2.0.4/src/iperf -c 10.10.10.2 -N > -t 40 > ------------------------------------------------------------ > Client connecting to 10.10.10.2, TCP port 5001 > TCP window size: 391 KByte (default) > ------------------------------------------------------------ > [ 3] local 10.10.10.3 port 56583 connected with 10.10.10.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-40.0 sec 29.3 GBytes 6.29 Gbits/sec > > Nicstat : henrik at naz01:/tmpfs# /export/home/henrik/nicstat -i ixgbe0 2 > Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util Sat > 21:13:02 ixgbe0 776175 1222.1 96592.9 18961.7 8228.4 66.00 63.7 83018.3 > 21:13:04 ixgbe0 773081 1217.2 96221.2 18885.3 8227.2 66.00 63.4 82717.5 > > To measure the NFS throughput over this link I have created a tmpfs > filesystem on the server to avoid the synchronous writes issue as much > as possible. > > Client : henrik at opensolaris:~# mount | grep /nfs > /nfs on 10.10.10.2:/tmpfs > remote/read/write/setuid/devices/forcedirectio/xattr/dev=4dc0007 on > Wed Aug 5 20:06:25 2009 > > Server : > henrik at naz01:/tmpfs# share | grep tmpfs > - /tmpfs sec=sys,root=10.10.10.3 "" > henrik at naz01:/tmpfs# mount | grep tmpfs > /tmpfs on swap read/write/setuid/devices/xattr/dev=4b80006 on Wed Aug > 5 21:59:31 2009 > > I have set the 'forcedirectio' option on the client mount to ensure that > the clients cache gets circumvented. > > Using the randomwrite microbenchmark in filebench ($filesize set to 1gb) > I get : > Local on tmpfs : > IO Summary: 5013937 ops, 82738.5 ops/s, (0/82738 r/w) 646.4mb/s, 71us > cpu/op, 0.0ms latency > > Tmpfs over NFS : > IO Summary: 383488 ops, 6328.2 ops/s, (0/6328 r/w) 49.4mb/s, 65us > cpu/op, 0.2ms latency > > These are 2 fully populated 4 socket machines - why the extremely low > transfer speed ? randomwrite.f is a single threaded workload (assuming you are using randomwrite.f filebench workload), which may not be sending enough work for the server to begin with. If you drive the number of threads in the workload higher (modify the nthreads variable in randomwrite.f), you should see better numbers, unless there is some other limits in the system. You can examine the CPU utilization of the client (and the server) machine to make sure that the client is busy sending work to the server.
-Piyush
