But it does not give you PIDs or user names? Or is there a way to find
these with standard lustre tools?
Michael
Am 11.02.2011 17:34, schrieb Andreas Dilger:
> On 2011-02-10, at 23:18, Michael Kluge wrote:
>> I am not aware of any possibility to map the current statistics in /proc
>> to UIDs. But I might be wrong. We had a script like this a while ago
>> which did not kill the I/O intensive processes but told us the PIDs.
>>
>> What we did is collecting for ~30 seconds the number of I/O operations
>> per node via /proc on all nodes. Then we attached an strace process to
>> each process on nodes with heavy I/O load. This strace intercepted only
>> the I/O calls and wrote one log file per process. If this strace is
>> running for the same amount of time for each process on a host, you just
>> need to sort the log files for size.
>
> On the OSS and MDS nodes there are per-client statistics that allow this kind
> of tracking. They can be seen in /proc/fs/lustre/obdfilter/*/exports/*/stats
> for detailed information (e.g. broken down by RPC type, bytes read/written),
> or /proc/fs/lustre/ost/OSS/*/req_history to just get a dump of the recent
> RPCs sent by each client.
>
> A little script was discussed in the thread "How to determine which lustre
> clients are loading filesystem" (2010-07-08):
>
>> Another way that I heard some sites were doing this is to use the "rpc
>> history". They may already have a script to do this, but the basics are
>> below:
>>
>> oss# lctl set_param ost.OSS.*.req_buffer_history_max=10240
>> {wait a few seconds to collect some history}
>> oss# lctl get_param ost.OSS.*.req_history
>>
>> This will give you a list of the past (up to) 10240 RPCs for the "ost_io"
>> RPC service, which is what you are observing the high load on:
>>
>> 3436037:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1340648957534353:448:Complete:1278612656:0s(-6s)
>> opc 3
>> 3436038:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1340648957536190:448:Complete:1278615489:1s(-41s)
>> opc 3
>> 3436039:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1340648957536193:448:Complete:1278615490:0s(-6s)
>> opc 3
>>
>> This output is in the format:
>>
>> identifier:target_nid:source_nid:rpc_xid:rpc_size:rpc_status:arrival_time:service_time(deadline)
>> opcode
>>
>> Using some shell scripting, one can find the clients sending the most RPC
>> requests:
>>
>> oss# lctl get_param ost.OSS.*.req_history | tr ":" " " | cut -d" " -f3,9,10
>> | sort | uniq -c | sort -nr | head -20
>>
>>
>> 3443 12345-192.168.20.159@tcp opc 3
>> 1215 12345-192.168.20.157@tcp opc 3
>> 121 12345-192.168.20.157@tcp opc 4
>>
>> This will give you a sorted list of the top 20 clients that are sending the
>> most RPCs to the ost and ost_io services, along with the operation being
>> done (3 = OST_READ, 4 = OST_WRITE, etc. see
>> lustre/include/lustre/lustre_idl.h).
>
>
>> Am Donnerstag, den 10.02.2011, 21:16 -0600 schrieb Satoshi Isono:
>>> Dear members,
>>>
>>> I am looking into the way which can detect userid or jobid on the Lustre
>>> client. Assumed the following condition;
>>>
>>> 1) Any users run any jobs through scheduler like PBS Pro, LSF or SGE.
>>> 2) A users processes occupy Lustre I/O.
>>> 3) Some Lustre servers (MDS?/OSS?) can detect high I/O stress on each
>>> server.
>>> 4) But Lustre server cannot make the mapping between jobid/userid and
>>> Lustre I/O processes having heavy stress, because there aren't userid on
>>> Lustre servers.
>>> 5) I expect that Lustre can monitor and can make the mapping.
>>> 6) If possible for (5), we can make a script which launches scheduler
>>> command like as qdel.
>>> 7) Heavy users job will be killed by job scheduler.
>>>
>>> I want (5) for Lustre capability, but I guess current Lustre 1.8 cannot
>>> perform (5). On the other hand, in order to map Lustre process to
>>> userid/jobid, are there any ways using like rpctrace or nid stats? Can you
>>> please your advice or comments?
>>>
>>> Regards,
>>> Satoshi Isono
>>> _______________________________________________
>>> Lustre-discuss mailing list
>>> [email protected]
>>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>>
>>
>> --
>>
>> Michael Kluge, M.Sc.
>>
>> Technische Universität Dresden
>> Center for Information Services and
>> High Performance Computing (ZIH)
>> D-01062 Dresden
>> Germany
>>
>> Contact:
>> Willersbau, Room A 208
>> Phone: (+49) 351 463-34217
>> Fax: (+49) 351 463-37773
>> e-mail: [email protected]
>> WWW: http://www.tu-dresden.de/zih
>> _______________________________________________
>> Lustre-discuss mailing list
>> [email protected]
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>
>
> Cheers, Andreas
> --
> Andreas Dilger
> Principal Engineer
> Whamcloud, Inc.
>
>
>
>
--
Michael Kluge, M.Sc.
Technische Universität Dresden
Center for Information Services and
High Performance Computing (ZIH)
D-01062 Dresden
Germany
Contact:
Willersbau, Room WIL A 208
Phone: (+49) 351 463-34217
Fax: (+49) 351 463-37773
e-mail: [email protected]
WWW: http://www.tu-dresden.de/zih
_______________________________________________
Lustre-discuss mailing list
[email protected]
http://lists.lustre.org/mailman/listinfo/lustre-discuss
