Found this too. iosnoop. Ported to linux a few years ago. might be useful.
http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html rgt Whitehead Network/System Administrator ----- On Sep 1, 2016, at 5:07 PM, Daniel Feenberg [email protected] wrote: > On Thu, 1 Sep 2016, Rob Taylor wrote: > >> Have you tried iotop? >> It will tell you what processes are moving the most disk io at any given >> instant. >> Still might not get you what you want, but it might make it easier to narrow >> down. > > iotop moves the sequential access processes to the top of the list, > because a proces doing sequential access processes more kilobytes/second > than one doing random access (because of cache hits, among other reasons). > Our problem program is not "top" in iotop. > > Actually, knowing the file name would probably be just as good as knowing > the process, since we could find the owner of the file and contact them. > > dan feenberg > >> >> rgt >> >> Whitehead Network/System Administrator >> >> ----- On Sep 1, 2016, at 3:05 PM, Daniel Feenberg [email protected] wrote: >> >>> Apparently heavy random I/O overloaded our fileserver last week, and >>> response was very slow. We solved the problem with additional spindles, >>> but we are curious to know which process is doing the random I/O. Perhaps >>> we could approach that user with an offer to help improve their turnaround >>> time by changing the code. Our users are mostly inexperienced students so >>> the possibility of suboptimal code is certainly there. Most usage is >>> sequential access to very large files that does not load the fileserver >>> much at all so this has been a new experience for us. >>> >>> We can easily track bytes/second but a process doing random I/O may use >>> very few bytes/second, but still occupy much of the fileservers capacity, >>> so it hasn't been fruitful to identify the processes doing the most reads >>> and writes. During the period of overload, few disks were showing more >>> than kilobytes/second of read or write, yet iostat revealed that several >>> disks were continuously at 100%. >>> >>> A program such as iostat will tell us which physical disk is busy, lsof >>> will tell us which file is open by which process, netstat and nfstat will >>> give aggregate statistics over all processes, but I can't find a program >>> that will tell us which process is occupying the fileservers attention >>> with expensive requests. >>> >>> We couldn't replace all the disks with SSD, but might be able to provide >>> SSD for some files, if we could identify the culprits. >>> >>> Daniel Feenberg >>> >>> _______________________________________________ >>> bblisa mailing list >>> [email protected] >>> http://www.bblisa.org/mailman/listinfo/bblisa _______________________________________________ bblisa mailing list [email protected] http://www.bblisa.org/mailman/listinfo/bblisa
