i am not quite sure, that you are hitting the recently fixed performance issue with ganesha with your scenario, as this was fixing a sendq issue.
it would be quite interesting to see
- which ganesha nfs version you are using. i think the fix went into the ibm028 level.
- netstat output during your tests, are there any findings on sendq values for the nfsd port ?
( like : netstat -an | grep 2049 | awk '{ print "SendQ: " $3 " Source: " $4 " Destination: " $5 }' | grep -v "SendQ: 0" )
Mit freundlichen Grüßen / Kind regards
Achim Rehor
| Software Technical Support Specialist AIX/ Emea HPC Support | |||
| IBM Certified Advanced Technical Expert - Power Systems with AIX | |||
| TSCC Software Service, Dept. 7922 | |||
| Global Technology Services | |||
| Phone: | +49-7034-274-7862 | IBM Deutschland | |
| E-Mail: | [email protected] | Am Weiher 24 | |
| 65451 Kelsterbach | |||
| Germany | |||
| IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Martin Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 WEEE-Reg.-Nr. DE 99369940 | |||
From:
"Alexander Saupp"
<[email protected]>
To:
[email protected]
Date:
15/10/2018 20:20
Subject:
[gpfsug-discuss]
Tuning: single client, single thread, small
files - native Scale vs NFS
Sent by:
[email protected]
Dear Spectrum Scale mailing list,
I'm part of IBM Lab Services - currently i'm having multiple customers asking me for optimization of a similar workloads.
The task is to tune a Spectrum Scale system (comprising ESS and CES protocol nodes) for the following workload:
A single Linux NFS client mounts an NFS export, extracts a flat tar archive with lots of ~5KB files.
I'm measuring the speed at which those 5KB files are written (`time tar xf archive.tar`).
I do understand that Spectrum Scale is not designed for such workload (single client, single thread, small files, single directory), and that such benchmark in not appropriate to benmark the system.
Yet I find myself explaining the performance for such scenario (git clone..) quite frequently, as customers insist that optimization of that scenario would impact individual users as it shows task duration.
I want to make sure that I have optimized the system as much as possible for the given workload, and that I have not overlooked something obvious.
When writing to GPFS directly I'm able to write ~1800 files / second in a test setup.
This is roughly the same on the protocol nodes (NSD client), as well as on the ESS IO nodes (NSD server).
When writing to the NFS export on the protocol node itself (to avoid any network effects) I'm only able to write ~230 files / second.
Writing to the NFS export from another node (now including network latency) gives me ~220 files / second.
There seems to be a huge performance degradation by adding NFS-Ganesha to the software stack alone. I wonder what can be done to minimize the impact.
- Ganesha doesn't seem to support 'async' or 'no_wdelay' options... anything equivalent available?
- Is there and expected advantage of using the network-latency tuned profile, as opposed to the ESS default throughput-performance profile?
- Are there other relevant Kernel params?
- Is there an expected advantage of raising the number of threads (NSD server (nsd*WorkerThreads) / NSD client (workerThreads) / Ganesha (NB_WORKER)) for the given workload (single client, single thread, small files)?
- Are there other relevant GPFS params?
- Impact of Sync replication, disk latency, etc is understood.
- I'm aware that 'the real thing' would be to work with larger files in a multithreaded manner from multiple nodes - and that this scenario will scale quite well.
I just want to ensure that I'm not missing something obvious over reiterating that massage to customers.
Any help was greatly appreciated - thanks much in advance!
Alexander Saupp
IBM Germany
Mit freundlichen Grüßen / Kind regards
Alexander Saupp
IBM Systems, Storage Platform, EMEA Storage Competence Center
| Phone: | +49 7034-643-1512 | IBM Deutschland GmbH | |
| Mobile: | +49-172 7251072 | Am Weiher 24 | |
| Email: | [email protected] | 65451 Kelsterbach | |
| Germany | |||
| IBM
Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Matthias Hartmann (Vorsitzender), Norbert Janzen, Stefan Lutz, Nicole Reimer, Dr. Klaus Seifert, Wolfgang Wendt Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 | |||
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
