Hello all;
Got a couple of Dell R720's running NexentaStor 3.1.4 on a 10GbE
network with jumbo frames enabled (MTU 9000).
iperf gives me over 9Gbps, so I'm confident the network itself is
capable.
We are dealing with a scenario where we need to move lots of data
around via NAS protocols, and right now I'm focused on NFS.
For example, we have a tree of ~10TB or so of relatively large files
(around 200MB each). I've made the following changes to the networking
stack on both server and client:
ndd -set /dev/tcp tcp_max_buf 2097152
ndd -set /dev/tcp tcp_cwnd_max 2097152
ndd -set /dev/tcp tcp_recv_hiwat 400000
ndd -set /dev/tcp tcp_xmit_hiwat 400000
On the client:
set ip:tcp_squeue_wput=1
set rpcmod:clnt_max_conns = 8
set nfs:nfs3_bsize=0x100000
set nfs:nfs3_max_transfer_size_cots=1048576
set nfs:nfs3_max_transfer_size=1048576
set nfs:nfs4_bsize=0x100000
set nfs:nfs4_max_transfer_size_cots=1048576
set nfs:nfs4_max_transfer_size=1048576
And am using the following options on the NFS client to mount the
export off the server:
mount -o wsize=1048576,rsize=1048576,vers=4 \
10.212.100.16:/volumes/datapool/DG_US_CONVERTED /mnt
I'm initiating a copy of the tree from the client by doing a good 'ol:
# cd /mnt/path
# tar cf - . | (cd /path/dst ; tar xvf -)
I'm only getting around 200MB/sec (~1.6Gbps). Does this sound like an
expected ceiling?
This is obviously only one thread of NFS. Should I be able to get
better throughput? Do I need to focus on ways to multiplex the
transfer? Are there any file copy tools that utilize multiple TCP
connections for data transport of a single data stream off my disk
(maybe lftp could work here).
Thanks for any thoughts.
Ray
_______________________________________________
nfs-discuss mailing list
[email protected]
