On 2011-02-10, at 23:18, Michael Kluge wrote:
> I am not aware of any possibility to map the current statistics in /proc
> to UIDs. But I might be wrong. We had a script like this a while ago
> which did not kill the I/O intensive processes but told us the PIDs.
>
> What we did is collecting for ~30 seconds the number of I/O operations
> per node via /proc on all nodes. Then we attached an strace process to
> each process on nodes with heavy I/O load. This strace intercepted only
> the I/O calls and wrote one log file per process. If this strace is
> running for the same amount of time for each process on a host, you just
> need to sort the log files for size.
On the OSS and MDS nodes there are per-client statistics that allow this kind
of tracking. They can be seen in /proc/fs/lustre/obdfilter/*/exports/*/stats
for detailed information (e.g. broken down by RPC type, bytes read/written), or
/proc/fs/lustre/ost/OSS/*/req_history to just get a dump of the recent RPCs
sent by each client.
A little script was discussed in the thread "How to determine which lustre
clients are loading filesystem" (2010-07-08):
> Another way that I heard some sites were doing this is to use the "rpc
> history". They may already have a script to do this, but the basics are
> below:
>
> oss# lctl set_param ost.OSS.*.req_buffer_history_max=10240
> {wait a few seconds to collect some history}
> oss# lctl get_param ost.OSS.*.req_history
>
> This will give you a list of the past (up to) 10240 RPCs for the "ost_io" RPC
> service, which is what you are observing the high load on:
>
> 3436037:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1340648957534353:448:Complete:1278612656:0s(-6s)
> opc 3
> 3436038:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1340648957536190:448:Complete:1278615489:1s(-41s)
> opc 3
> 3436039:192.168.20.1@tcp:12345-192.168.20.159@tcp:x1340648957536193:448:Complete:1278615490:0s(-6s)
> opc 3
>
> This output is in the format:
>
> identifier:target_nid:source_nid:rpc_xid:rpc_size:rpc_status:arrival_time:service_time(deadline)
> opcode
>
> Using some shell scripting, one can find the clients sending the most RPC
> requests:
>
> oss# lctl get_param ost.OSS.*.req_history | tr ":" " " | cut -d" " -f3,9,10 |
> sort | uniq -c | sort -nr | head -20
>
>
> 3443 12345-192.168.20.159@tcp opc 3
> 1215 12345-192.168.20.157@tcp opc 3
> 121 12345-192.168.20.157@tcp opc 4
>
> This will give you a sorted list of the top 20 clients that are sending the
> most RPCs to the ost and ost_io services, along with the operation being done
> (3 = OST_READ, 4 = OST_WRITE, etc. see lustre/include/lustre/lustre_idl.h).
> Am Donnerstag, den 10.02.2011, 21:16 -0600 schrieb Satoshi Isono:
>> Dear members,
>>
>> I am looking into the way which can detect userid or jobid on the Lustre
>> client. Assumed the following condition;
>>
>> 1) Any users run any jobs through scheduler like PBS Pro, LSF or SGE.
>> 2) A users processes occupy Lustre I/O.
>> 3) Some Lustre servers (MDS?/OSS?) can detect high I/O stress on each server.
>> 4) But Lustre server cannot make the mapping between jobid/userid and Lustre
>> I/O processes having heavy stress, because there aren't userid on Lustre
>> servers.
>> 5) I expect that Lustre can monitor and can make the mapping.
>> 6) If possible for (5), we can make a script which launches scheduler
>> command like as qdel.
>> 7) Heavy users job will be killed by job scheduler.
>>
>> I want (5) for Lustre capability, but I guess current Lustre 1.8 cannot
>> perform (5). On the other hand, in order to map Lustre process to
>> userid/jobid, are there any ways using like rpctrace or nid stats? Can you
>> please your advice or comments?
>>
>> Regards,
>> Satoshi Isono
>> _______________________________________________
>> Lustre-discuss mailing list
>> [email protected]
>> http://lists.lustre.org/mailman/listinfo/lustre-discuss
>>
>
> --
>
> Michael Kluge, M.Sc.
>
> Technische Universität Dresden
> Center for Information Services and
> High Performance Computing (ZIH)
> D-01062 Dresden
> Germany
>
> Contact:
> Willersbau, Room A 208
> Phone: (+49) 351 463-34217
> Fax: (+49) 351 463-37773
> e-mail: [email protected]
> WWW: http://www.tu-dresden.de/zih
> _______________________________________________
> Lustre-discuss mailing list
> [email protected]
> http://lists.lustre.org/mailman/listinfo/lustre-discuss
Cheers, Andreas
--
Andreas Dilger
Principal Engineer
Whamcloud, Inc.
_______________________________________________
Lustre-discuss mailing list
[email protected]
http://lists.lustre.org/mailman/listinfo/lustre-discuss
