[SLUG] Re: Linux Disk Performance/File IO per process
On Mon, 29 Jan 2001, Szabolcs Szakacsits wrote: > On Mon, 29 Jan 2001, Chris Evans wrote: > > > Stephen Tweedie has a rather funky i/o stats enhancement patch which > > should provide what you need. It comes with RedHat7.0 and gives decent > > disk statistics in /proc/partitions. > > Monitoring via /proc [not just IO but close to anything] has the > features: > - slow, not atomic, not scalable > - if kernel decides explicitely or due to a "bug" to refuse doing >IO, you get something like this [even using a mlocked, RT monitor], >procsmemoryswap io system cpu > r b w swpd free buff cache si sobibo incs us sy id > 0 1 1 27116 1048 736 152832 128 1972 2544 869 44 1812 2 43 55 > 5 0 2 27768 1048 744 153372 52 1308 2668 777 43 1772 2 61 37 > 0 2 1 28360 1048 752 153900 332 564 2311 955 49 2081 1 68 31 > > 1 7 2 28356 1048 752 153708 3936 0 2175 29091 494 27348 0 1 99 > 1 0 2 28356 1048 792 153656 172 0 7166 0 144 838 4 17 80 > > In short, monitoring via /proc is unreliable. Not really unreliable, but definitely with _serious_ latency issues :) due to taking the mmap_sem. Acquiring the mmap_sem semaphore can take a really long time under load.. and sys_brk downs this semaphore first thing, as does task_mem() and proc_pid_stat()... If someone has the mmap_sem you want, and is pushing disk I/O when that disk is saturated, you are in for a long wait. This I think is what you see with your mlocked RT monitor (pretty similar to my mlocked RT monitor I suspect) In fact, that darn monitor can have a decidedly negative impact on system performance because it can take an arbitrary task's mana connection and then fault while throttling it... I think ;-) -Mike -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] RE: Linux Disk Performance/File IO per process
On Mon, 29 Jan 2001 [EMAIL PROTECTED] wrote: > Thanks to both Jens and Chris - this provides the information I need to > obtain our busy rate > It's unfortunate that the kernel needs to be patched to provide this > information - hopefully it will become part of the kernel soon. > > I had a response saying that this shouldn't become part of the kernel due to > the performance cost that obtaining such data will involve. I agree that a > cost is involved here, however I think it's up to the user to decide which > cost is more expensive to them - getting the data, or not being able to see > how busy their disks are. My feeling here is that this support could be user > configurable at run time - eg 'cat 1 > /proc/getdiskperf'. Hi, I disagree with this runtime variable. It is unnecessary complexity. Maintaining a few counts is total noise compared with the time I/O takes. Cheers Chris -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] RE: Linux Disk Performance/File IO per process
On Mon, 29 Jan 2001 [EMAIL PROTECTED] wrote: Thanks to both Jens and Chris - this provides the information I need to obtain our busy rate It's unfortunate that the kernel needs to be patched to provide this information - hopefully it will become part of the kernel soon. I had a response saying that this shouldn't become part of the kernel due to the performance cost that obtaining such data will involve. I agree that a cost is involved here, however I think it's up to the user to decide which cost is more expensive to them - getting the data, or not being able to see how busy their disks are. My feeling here is that this support could be user configurable at run time - eg 'cat 1 /proc/getdiskperf'. Hi, I disagree with this runtime variable. It is unnecessary complexity. Maintaining a few counts is total noise compared with the time I/O takes. Cheers Chris -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] Re: Linux Disk Performance/File IO per process
On Mon, 29 Jan 2001, Szabolcs Szakacsits wrote: On Mon, 29 Jan 2001, Chris Evans wrote: Stephen Tweedie has a rather funky i/o stats enhancement patch which should provide what you need. It comes with RedHat7.0 and gives decent disk statistics in /proc/partitions. Monitoring via /proc [not just IO but close to anything] has the features: - slow, not atomic, not scalable - if kernel decides explicitely or due to a "bug" to refuse doing IO, you get something like this [even using a mlocked, RT monitor], procsmemoryswap io system cpu r b w swpd free buff cache si sobibo incs us sy id 0 1 1 27116 1048 736 152832 128 1972 2544 869 44 1812 2 43 55 5 0 2 27768 1048 744 153372 52 1308 2668 777 43 1772 2 61 37 0 2 1 28360 1048 752 153900 332 564 2311 955 49 2081 1 68 31 frozen 1 7 2 28356 1048 752 153708 3936 0 2175 29091 494 27348 0 1 99 1 0 2 28356 1048 792 153656 172 0 7166 0 144 838 4 17 80 In short, monitoring via /proc is unreliable. Not really unreliable, but definitely with _serious_ latency issues :) due to taking the mmap_sem. Acquiring the mmap_sem semaphore can take a really long time under load.. and sys_brk downs this semaphore first thing, as does task_mem() and proc_pid_stat()... If someone has the mmap_sem you want, and is pushing disk I/O when that disk is saturated, you are in for a long wait. This I think is what you see with your mlocked RT monitor (pretty similar to my mlocked RT monitor I suspect) In fact, that darn monitor can have a decidedly negative impact on system performance because it can take an arbitrary task's mana connection and then fault while throttling it... I think ;-) -Mike -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] RE: Linux Disk Performance/File IO per process
> -Original Message- > From: Chris Evans [mailto:[EMAIL PROTECTED]] > Sent: Monday, 29 January 2001 13:04 > To: Tony Young > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: Re: Linux Disk Performance/File IO per process > > > > On Mon, 29 Jan 2001 [EMAIL PROTECTED] wrote: > > > All, > > > > I work for a company that develops a systems and > performance management > > product for Unix (as well as PC and TANDEM) called > PROGNOSIS. Currently we > > support AIX, HP, Solaris, UnixWare, IRIX, and Linux. > > > > I've hit a bit of a wall trying to expand the data provided > by our Linux > > solution - I can't seem to find anywhere that provides the > metrics needed to > > calculate disk busy in the kernel! This is a major piece of > information that > > any mission critical system administrator needs to > successfully monitor > > their systems. > > Stephen Tweedie has a rather funky i/o stats enhancement patch which > should provide what you need. It comes with RedHat7.0 and gives decent > disk statistics in /proc/partitions. > > Unfortunately this patch is not yet in the 2.2 or 2.4 kernel. > I'd like to > see it make the kernel as a 2.4.x item. Failing that, it'll > probably make > the 2.5 kernel. > > Cheers > Chris > Thanks to both Jens and Chris - this provides the information I need to obtain our busy rate It's unfortunate that the kernel needs to be patched to provide this information - hopefully it will become part of the kernel soon. I had a response saying that this shouldn't become part of the kernel due to the performance cost that obtaining such data will involve. I agree that a cost is involved here, however I think it's up to the user to decide which cost is more expensive to them - getting the data, or not being able to see how busy their disks are. My feeling here is that this support could be user configurable at run time - eg 'cat 1 > /proc/getdiskperf'. Thanks for your quick responses. Tony... -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] Re: Linux Disk Performance/File IO per process
On Mon, Jan 29 2001, [EMAIL PROTECTED] wrote: > All, > > I work for a company that develops a systems and performance management > product for Unix (as well as PC and TANDEM) called PROGNOSIS. Currently we > support AIX, HP, Solaris, UnixWare, IRIX, and Linux. > > I've hit a bit of a wall trying to expand the data provided by our Linux > solution - I can't seem to find anywhere that provides the metrics needed to > calculate disk busy in the kernel! This is a major piece of information that > any mission critical system administrator needs to successfully monitor > their systems. The stock kernel doesn't provide either, but at least with Stephen's sard patches you can get system wide I/O metrics. ftp.linux.org.uk/pub/linux/sct/fs/profiling -- Jens Axboe -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] Re: Linux Disk Performance/File IO per process
On Mon, Jan 29 2001, [EMAIL PROTECTED] wrote: All, I work for a company that develops a systems and performance management product for Unix (as well as PC and TANDEM) called PROGNOSIS. Currently we support AIX, HP, Solaris, UnixWare, IRIX, and Linux. I've hit a bit of a wall trying to expand the data provided by our Linux solution - I can't seem to find anywhere that provides the metrics needed to calculate disk busy in the kernel! This is a major piece of information that any mission critical system administrator needs to successfully monitor their systems. The stock kernel doesn't provide either, but at least with Stephen's sard patches you can get system wide I/O metrics. ftp.linux.org.uk/pub/linux/sct/fs/profiling -- Jens Axboe -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
[SLUG] RE: Linux Disk Performance/File IO per process
-Original Message- From: Chris Evans [mailto:[EMAIL PROTECTED]] Sent: Monday, 29 January 2001 13:04 To: Tony Young Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: Linux Disk Performance/File IO per process On Mon, 29 Jan 2001 [EMAIL PROTECTED] wrote: All, I work for a company that develops a systems and performance management product for Unix (as well as PC and TANDEM) called PROGNOSIS. Currently we support AIX, HP, Solaris, UnixWare, IRIX, and Linux. I've hit a bit of a wall trying to expand the data provided by our Linux solution - I can't seem to find anywhere that provides the metrics needed to calculate disk busy in the kernel! This is a major piece of information that any mission critical system administrator needs to successfully monitor their systems. Stephen Tweedie has a rather funky i/o stats enhancement patch which should provide what you need. It comes with RedHat7.0 and gives decent disk statistics in /proc/partitions. Unfortunately this patch is not yet in the 2.2 or 2.4 kernel. I'd like to see it make the kernel as a 2.4.x item. Failing that, it'll probably make the 2.5 kernel. Cheers Chris Thanks to both Jens and Chris - this provides the information I need to obtain our busy rate It's unfortunate that the kernel needs to be patched to provide this information - hopefully it will become part of the kernel soon. I had a response saying that this shouldn't become part of the kernel due to the performance cost that obtaining such data will involve. I agree that a cost is involved here, however I think it's up to the user to decide which cost is more expensive to them - getting the data, or not being able to see how busy their disks are. My feeling here is that this support could be user configurable at run time - eg 'cat 1 /proc/getdiskperf'. Thanks for your quick responses. Tony... -- SLUG - Sydney Linux User Group Mailing List - http://slug.org.au/ More Info: http://slug.org.au/lists/listinfo/slug
