Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average

2020-06-04 Thread Saula, Oluwasijibomi
Stephen, Looked into client requests, and it doesn't seem to lean heavily on any one NSD server. Of course, this is an eyeball assessment after reviewing IO request percentages to the different NSD servers from just a few nodes. By the way, I later discovered our TSM/NSD server couldn't handle

Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average

2020-06-04 Thread Kumaran Rajaram
Hi, >> I do notice nsd03/nsd04 have long waiters, but nsd01 doesn't (nsd02-ib is offline for now): Please issue "mmlsdisk -m" in NSD client to ascertain the active NSD server serving a NSD. Since nsd02-ib is offlined, it is possible that some servers would be serving higher NSDs than the

Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average

2020-06-04 Thread Valdis Klētnieks
On Thu, 04 Jun 2020 15:33:18 -, "Saula, Oluwasijibomi" said: > However, I still can't understand why write IO operations are 5x more latent > than ready operations to the same class of disks. Two things that may be biting you: First, on a RAID 5 or 6 LUN, most of the time you only need to

Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average

2020-06-04 Thread Stephen Ulmer
Note that if nsd02-ib is offline, that nsd03-ib is now servicing all of the NSDs for *both* servers, and that if nsd03-ib gets busy enough to appear offline, then nsd04-ib would be next in line to get the load of all 3. The two servers with the problems are in line after the one that is off.

Re: [gpfsug-discuss] Client Latency and High NSD Server Load Average

2020-06-04 Thread Frederick Stock
From the waiters you provided I would guess there is something amiss with some of your storage systems.  Since those waiters are on NSD servers they are waiting for IO requests to the kernel to complete.  Generally IOs are expected to complete in milliseconds, not seconds.  You could look at the