On Thu, Aug 6, 2009 at 3:52 PM, Dai Ngo<Dai.Ngo at sun.com> wrote: > Henrik Johansen wrote: >> >> [Dai Ngo] wrote: >>> >>> Henrik Johansen wrote: >>>> >>>> Henrik Johansen wrote: >>>>> >>>>> Piyush Shivam wrote: >>>>>> >>>>>> On 08/05/09 15:53, Henrik Johansen wrote: >>>>>>> >>>>>>> Hi list, >>>>>>> >>>>>>> I have 2 servers which are directly connected via ixgbe based nics, >>>>>>> both >>>>>>> running OpenSolaris 2009.06. >>>>>>> >>>>>>> The actual network connection seems fine, iperf reports ~6.3 >>>>>>> Gbits/sec >>>>>>> in terms of throughput and nicstat seems to agree that the nics are >>>>>>> ~63% >>>>>>> utilized. >>>>>>> Iperf : henrik at opensolaris:~# ./iperf-2.0.4/src/iperf -c 10.10.10.2 >>>>>>> -N -t 40 >>>>>>> ------------------------------------------------------------ >>>>>>> Client connecting to 10.10.10.2, TCP port 5001 >>>>>>> TCP window size: 391 KByte (default) >>>>>>> ------------------------------------------------------------ >>> >>> Can you verify the TCP window size on both client and server system with >>> this command: >>> >>> # ndd ?-get ?/dev/tcp ?tcp_xmit_hiwat >>> >>> # ndd ? -get ?/dev/tcp ?tcp_recv_hiwat >> >> Both client and server have both set to 400000. >> > Can you try 1024000, this is the default in snv_119. > Did you verify the new setting take effect on NFS connections?
I'm *so* glad to see that this is finally getting some attention. Are there any good reasons not to increase them to that value on S10 - snv_118? > > -Dai On Solaris 10 (vintage October 2007) I noticed that the following tuning was helpful on gigabit networks. The part that is probably interesting to you is nfs3_bsize, which had a clearly inadequate default on for 1 GigE. I'm not sure that the transfers really ever got up to the size I tuned it to (suspect it was choked elsewhere) but I was happy with the improvement I saw. == Begin quote from my notes == Details: Simple single-stream TCP tests on a T2000 show that increasing the xmit & recv hiwat values increases throughput by nearly 2x (500 Mb/s -> 904 Mb/s). On top of that, tuning the nfs3 block size to 1 MB increases large NFS read performance by 3.6x (243 Mb/s -> 872 Mb/s). Details: Simple single-stream TCP tests on a T2000 show that increasing the xmit & recv hiwat values increases throughput by nearly 2x (500 Mb/s -> 904 Mb/s). On top of that, tuning the nfs3 block size to 1 MB increases large NFS read performance by 3.6x (243 Mb/s -> 872 Mb/s). ndd -set /dev/tcp tcp_xmit_hiwat 131072 ndd -set /dev/tcp tcp_recv_hiwat 131072 Add to /etc/system: * NFS Client tuning - do 1 MByte transfers by default * T2000 NFS read performance jumps by 3.6x set nfs:nfs3_bsize=0x100000 * END NFS client tuning == End quote from my notes == I don't know if there is similar tuning for NFSv4. -- Mike Gerdts http://mgerdts.blogspot.com/
