Apologies for being terse, in a hurry here. 1) -o async doesn't work with NFS, don't use that. 2) how big are the text versus binary files? 3) how are you copying them over nfs?
I suspect, (could be wrong of course) that the ascii files are a lot smaller than the binary files, so what's happening is that for binary files, the client is issuing write-behind async, however for ascii files its issuing the writes at close time which will force the sync flag. -Alfred * Brent Jones <[email protected]> [090128 19:38] wrote: > Hello FreeBSD users, > I am running into some performance problems with NFSv3/v4 mounts. > I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares > The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram > and a FreeBSD 7.1 32bit with same hardware. > > The issue I am seeing, is that for certain file types, the FreeBSD NFS > client will either issue an ASYNC write, or an FSYNC. > However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP > versions of the protocol, so that should be the default. > Issuing FSYNC's for every compete block transmitted adds substantial > overhead and slows everything down. > > The two test files I have that can reproduce this data are a file > created by 'dump' which is just binary data: > > $ file testbinery > testbinery: data > > ASCII text file from a Maildir format: > > $ file ascittest > ascittest: ASCII mail text > > My NFS mount command lines I have tried to get all data to ASYNC write: > > $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > > Here is an excerpt from a snoop from the binary data file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 > pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > > > And on and on.. it will acheive near full wire-speed, about 110MB/sec > during the copy > > > Here is the same snoop, only copying the ASCII mail file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest > pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > > > And so on. I've reproduced this with several files, and the only > difference between tests is the file type. > Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the > file type/contents? > If so, is there a tuneable setting to make all write ASYNC? > Otherwise, FSYNC'ing for every block written over NFS will cause so > many IOPS on the NFS server, that performance will degrade severely. > > Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for > any file type, if mounted with NFSv3 of NFSv4 (TCP). > > Any ideas? > > Thanks in advance! > > -- > Brent Jones > [email protected] > _______________________________________________ > [email protected] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "[email protected]" -- - Alfred Perlstein _______________________________________________ [email protected] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
