Hello, Oluwasijibomi , I suppose you are not running ESS (might be wrong on this).
I'd check the IO history on the NSD servers (high IO times?) and in addition the IO traffic at the block device level , e.g. with iostat or the like (still high IO times there? Are the IO sizes ok or too low on the NSD servers with high write latencies? ). What's the picture on your storage back-end? All caches active? Is the storage backend fully loaded or rather idle? How is storage connected? SAS? FC? IB? What is the actual IO pattern when you see these high latencies? Do you run additional apps on some or all of youre NSD servers? Mit freundlichen Grüßen / Kind regards Dr. Uwe Falke IT Specialist Global Technology Services / Project Services Delivery / High Performance Computing +49 175 575 2877 Mobile Rathausstr. 7, 09111 Chemnitz, Germany uwefa...@de.ibm.com IBM Services IBM Data Privacy Statement IBM Deutschland Business & Technology Services GmbH Geschäftsführung: Dr. Thomas Wolter, Sven Schooss Sitz der Gesellschaft: Ehningen Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Saula, Oluwasijibomi" <oluwasijibomi.sa...@ndsu.edu> To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org> Date: 03/06/2020 23:45 Subject: [EXTERNAL] [gpfsug-discuss] Client Latency and High NSD Server Load Average Sent by: gpfsug-discuss-boun...@spectrumscale.org Hello, Anyone faced a situation where a majority of NSDs have a high load average and a minority don't? Also, is 10x NSD server latency for write operations than for read operations expected in any circumstance? We are seeing client latency between 6 and 9 seconds and are wondering if some GPFS configuration or NSD server condition may be triggering this poor performance. Thanks, Oluwasijibomi (Siji) Saula HPC Systems Administrator / Information Technology Research 2 Building 220B / Fargo ND 58108-6050 p: 701.231.7749 / www.ndsu.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=fTuVGtgq6A14KiNeaGfNZzOOgtHW5Lm4crZU6lJxtB8&m=ql8z1YSfrzUgT8kXQBMEUuA8uyuprz6-fpvC660vG5A&s=JSYPIzNMZFNp17VaqcNWNuwwUE_nQMKu47mOOUonLp0&e= _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss