Hi Bernd, On 02/09/10 17:21, Bernd Schubert wrote: > On Thursday, September 02, 2010, Andreas Dilger wrote: >> On 2010-09-02, at 06:43, Tina Friedrich wrote: >>> Causing most grieve at the moment is that we sometimes see delays >>> writing files. From the writing clients end, it simply looks as if I/O >>> stops for a while (we've seen 'pauses' of anything up to 10 seconds). >>> This appears to be independent of what client does the writing, and >>> software doing the writing. We investigated this a bit using strace and >>> dd; the 'slow' calls appear to always be either open, write, or close >>> calls. Usually, these take well below 0.001s; in around 0.5% or 1% of >>> cases, they take up to multiple seconds. It does not seem to be >>> associated with any specific OST, OSS, client or anything; there is >>> nothing in any log files or any exceptional load on MDS or OSS or any of >>> the clients. >> >> This is most likely associated with delays in committing the journal on the >> MDT or OST, which can happen if the journal fills completely. Having >> larger journals can help, if you have enough RAM to keep them all in >> memory and not overflow. Alternately, if you make the journals small it >> will limit the latency, at the cost of reducing overall performance. A >> third alternative might be to use SSDs for the journal devices. > > As diamond uses DDN hardware, it should help in general with performance to > update to 1.8 and to enable the async journal feature. I guess it also might > help to reduce those delays, as writes are more optimized.
We have been considering an update; however, due to to this being a production file system (and an important one), that's not something that we can do easily. > A question, though. Tina, do you use our ddn udev rules, which tune the > devices for optimized performance? If not, please send a mail to > [email protected] and ask for a recent udev rpm please (available for RHEL5 only > so far, also *might* work on SLES11, but udev syntax changes to often, IMHO). > And put [lustre] into the subject line please, as the lustre team maintains > them. Well; I don't think so; not 100% sure. There does not appear to be anything DDN specific in our udev rules (which makes me think that's a 'no'). I have sent an email requesting them, and shall look into that, as well. Tina -- Tina Friedrich, Computer Systems Administrator, Diamond Light Source Ltd Diamond House, Harwell Science and Innovation Campus - 01235 77 8442 _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
