Hi Kevin, Just going out on a very weird limb here...but you're not by chance seeing this behavior on DDN hardware that runs the SFA OS are you? (e.g. SFA12K, 7K, 14K, etc.) We just started seeing some very weird and high latency on some of our SFA12ks (that have otherwise been solid both in terms of stability and performance) but only on certain volumes and the affected volumes change. It's very bizzarre and we've been working closely with DDN to track down the root cause but we've not yet found a smoking gun. The timing and description of your problem sounded eerily similar to what we're seeing so I'd thought I'd ask.
-Aaron -- Aaron Knister NASA Center for Climate Simulation (Code 606.2) Goddard Space Flight Center (301) 286-2776 On Tue, 3 Jul 2018, Buterbaugh, Kevin L wrote:
Hi all, We are experiencing some high I/O wait times (5 - 20 seconds!) on some of our NSDs as reported by “mmdiag —iohist" and are struggling to understand why. One of the confusing things is that, while certain NSDs tend to show the problem more than others, the problem is not consistent … i.e. the problem tends to move around from NSD to NSD (and storage array to storage array) whenever we check … which is sometimes just a few minutes apart. In the past when I have seen “mmdiag —iohist” report high wait times like this it has *always* been hardware related. In our environment, the most common cause has been a battery backup unit on a storage array controller going bad and the storage array switching to write straight to disk. But that’s *not* happening this time. Is there anything within GPFS / outside of a hardware issue that I should be looking for?? Thanks! — Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education [email protected] - (615)875-9633
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
