All NFS IO requires syncing. The client does send explicit fsync (commit). If the NFS server does not sync, a server fail will cause data loss! (for small files <1M it really does not matter if it is sync on write or sync on close/explicit commit)
while that may be ok for a "git pull" or similar, in general it violates the NFS spec. The client can decide to cache, and usually NFSv4 does less caching (for better consistency) So the observed factor 100 is realistic. Latencies will make matters worse, so the FS should be tuned for very small random IO (small blocksize - small subblock-size will not help) If you were to put the Linux kernel NFS server into the picture, it will behave very much the same - although Ganesha could be a bit more efficient (by some percent - certainly less then 200%). But hey - this is a GPFS cluster not some NAS box. Run "git pull" on tthe GPFS client. Enjoy the 1800 files/sec (or more). Modify the files on your XY client mounting over NFS. Use a wrapper script to automatically have your AD or LDAP user id SSH into the cluster to perform it. Michael Mit freundlichen Grüßen / with best regards Michael Diederich IBM Systems Group Spectrum Scale Software Development Contact Information IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294 mail: fon: address: michael.dieder...@de.ibm.com +49-7034-274-4062 Am Weiher 24 D-65451 Kelsterbach From: valdis.kletni...@vt.edu To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> Cc: Silvana De Gyves <silva...@mx1.ibm.com>, Jay Vaddi <jayva...@us.ibm.com>, Michael Diederich <dieder...@de.ibm.com> Date: 10/16/2018 02:42 AM Subject: Re: [gpfsug-discuss] Tuning: single client, single thread, small files - native Scale vs NFS Sent by: Valdis Kletnieks <val...@vt.edu> On Mon, 15 Oct 2018 18:34:50 -0400, "Kumaran Rajaram" said: > 1. >>When writing to GPFS directly I'm able to write ~1800 files / second in a test setup. > >>This is roughly the same on the protocol nodes (NSD client), as well as > on the ESS IO nodes (NSD server). > > 2. >> When writing to the NFS export on the protocol node itself (to avoid > any network effects) I'm only able to write ~230 files / second. > IMHO #2, writing to the NFS export on the protocol node should be same as #1. > Protocol node is also a NSD client and when you write from a protocol node, it > will use the NSD protocol to write to the ESS IO nodes. In #1, you cite seeing > ~1800 files from protocol node and in #2 you cite seeing ~230 file/sec which > seem to contradict each other. I think he means this: 1) ssh nsd_server 2) cd /gpfs/filesystem/testarea 3) (whomp out 1800 files/sec) 4) mount -t nfs localhost:/gpfs/filesystem/testarea /mnt/test 5) cd /mnt/test 6) Watch the same test struggle to hit 230. Indicating the issue is going from NFS to GPFS (For what it's worth, we've had issues with Ganesha as well...) [attachment "att4z9wh.dat" deleted by Michael Diederich/Germany/IBM]
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss