Hi Adrian, thanks for sharing these with us.
Adrian Ulrich wrote: > I uploaded two lustre-related modules to the CPAN: > > #1: Lustre::Info provides easy access to information located > at /proc/fs/lustre, it also comes with a 'performance monitoring' > script called 'lustre-info.pl' I did have a bit of a play with the lustre-info.pl script on our test file system and it seems to work nicely. If you've got a lot of OSTs on your server you need a wide monitor for some of the options like --monitor=ost-patterns for all OSTs... We are currently running Lustre 1.6.7.2 (+ a few patches) on our OSTs, in case this makes a difference for my issues below. [snip] > Examples and details: > > Lustre::Info and lustre-info.pl > --------------------------------------- [snip] > The module also includes a script called 'lustre-info.pl' that can > be used to gather some live performance statistics: > > Use `--ost-stats' to get a quick overview on what's going on: > $ lustre-info.pl --ost-stats In our case this looks like this (on a very quiet file system): > play01-OST0000 (@ /dev/sdb) : write= 0.000 MB/s, read= 0.000 MB/s, > create= 0.0 R/s, destroy= 0.0 R/s, setattr= 0.0 R/s, preprw= 0.0 R/s > play01-OST0001 (@ /dev/sdc) : write= 0.000 MB/s, read= 0.000 MB/s, > create= 0.0 R/s, destroy= 0.0 R/sUse of uninitialized value in division (/) > at /usr/local/bin/lustre-info.pl line 187. > , setattr= 0.0 R/s, preprw= 0.0 R/s > play01-OST0002 (@ /dev/sdd) : write= 0.000 MB/s, read= 0.000 MB/s, > create= 0.0 R/s, destroy= 0.0 R/s, setattr= 0.0 R/s, preprw= 0.0 R/s > play01-OST0003 (@ /dev/sde) : write= 0.000 MB/s, read= 0.000 MB/s, > create= 0.0 R/s, destroy= 0.0 R/s, setattr= 0.0 R/s, preprw= 0.0 R/s > play01-OST0004 (@ /dev/sdf) : write= 0.000 MB/s, read= 0.000 MB/s, > create= 0.0 R/s, destroy= 0.0 R/sUse of uninitialized value in division (/) > at /usr/local/bin/lustre-info.pl line 187. > , setattr= 0.0 R/s, preprw= 0.0 R/s > play01-OST0005 (@ /dev/sdg) : write= 0.000 MB/s, read= 0.000 MB/s, > create= 0.0 R/s, destroy= 0.0 R/sUse of uninitialized value in division (/) > at /usr/local/bin/lustre-info.pl line 187. > , setattr= 0.0 R/s, preprw= 0.0 R/s Note the 'Use of uninitialized value in division...' errors. Looking at the code it seems the value for 'setattr' is missing from the stats file for some of our OSTs. Looking at the stats file, indeed the setattr line is missing for some OSTs. Has anyone seen this before? What could have caused this? > You can also get client<->ost details via `--monitor=MODE' > > $ lustre-info.pl --monitor=ost --as-list # this will only show clients where > read+write >= 1MB/s >> client nid | lustre1-OST0006 | lustre1-OST000e | lustre1-OST0016 >> | lustre1-OST001e | +++ TOTALS +++ (MB/s) > 10.201.46...@o2ib | r= 0.0, w= 0.0 | r= 0.0, w= 0.0 | r= 0.0, w= > 0.0 | r= 0.0, w= 1.1 | read= 0.0, write= 1.1 > 10.201.47...@o2ib | r= 0.0, w= 0.0 | r= 0.0, w= 1.2 | r= 0.0, w= > 2.0 | r= 0.0, w= 0.0 | read= 0.0, write= 3.2 'lustre-info.pl --monitor=io-size' seems to sit at "collecting data, please wait..." for a very long time until I killed it, I have not had the time to debug this yet. Kind regards, Frederik -- Frederik Ferner Computer Systems Administrator phone: +44 1235 77 8624 Diamond Light Source Ltd. mob: +44 7917 08 5110 (Apologies in advance for the lines below. Some bits are a legal requirement and I have no control over them.) _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
