Re: Disk IO issues

2009-01-22 Thread Mike McGrath
On Wed, 21 Jan 2009, Ray Van Dolson wrote: On Mon, Jan 19, 2009 at 10:55:35AM -0600, Mike McGrath wrote: On Mon, 19 Jan 2009, Jesse Keating wrote: On Mon, 2009-01-19 at 10:02 -0600, Mike McGrath wrote: on the netapp. Er, this is on nfs1 right, not the netapp? My

Re: Disk IO issues

2009-01-21 Thread Ray Van Dolson
On Mon, Jan 19, 2009 at 10:55:35AM -0600, Mike McGrath wrote: On Mon, 19 Jan 2009, Jesse Keating wrote: On Mon, 2009-01-19 at 10:02 -0600, Mike McGrath wrote: on the netapp. Er, this is on nfs1 right, not the netapp? My mistake, correct. All this is on nfs1 which has directly

Re: Disk IO issues

2009-01-19 Thread Mike McGrath
On Wed, 31 Dec 2008, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At least I think we are. This is

Re: Disk IO issues

2009-01-19 Thread Mike McGrath
On Mon, 19 Jan 2009, Mike McGrath wrote: The big changers? 1) Better use of the share in our scripts. 2) A larger readahead value (blockdev) I forgot one more big change, kojipkgs (the web server our builders use to get the packages off the nfs share) now has a squid server on it. Instead

Re: Disk IO issues

2009-01-19 Thread Jesse Keating
On Mon, 2009-01-19 at 10:02 -0600, Mike McGrath wrote: on the netapp. Er, this is on nfs1 right, not the netapp? -- Jesse Keating Fedora -- FreedomĀ² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part

Re: Disk IO issues

2009-01-19 Thread Mike McGrath
On Mon, 19 Jan 2009, Jesse Keating wrote: On Mon, 2009-01-19 at 10:02 -0600, Mike McGrath wrote: on the netapp. Er, this is on nfs1 right, not the netapp? My mistake, correct. All this is on nfs1 which has directly attached storage. -Mike

Re: Disk IO issues

2009-01-04 Thread Mike McGrath
On Sat, 3 Jan 2009, Matt Domsch wrote: On Sun, Jan 04, 2009 at 03:02:55AM +, Kostas Georgiou wrote: On Sat, Jan 03, 2009 at 06:32:38PM -0600, Matt Domsch wrote: What ext3 journaling options are enabled (e.g. what does 'mount' say)? If it's data=ordered (the default), that's OK. If

Re: Disk IO issues

2009-01-03 Thread Matt Domsch
On Wed, Dec 31, 2008 at 02:42:27PM -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At least I

Re: Disk IO issues

2009-01-03 Thread Kostas Georgiou
On Sat, Jan 03, 2009 at 06:32:38PM -0600, Matt Domsch wrote: What ext3 journaling options are enabled (e.g. what does 'mount' say)? If it's data=ordered (the default), that's OK. If it's data=journal, then all the data gets written twice (first to the journal, then the journal to the disk),

Re: Disk IO issues

2009-01-03 Thread Matt Domsch
On Sun, Jan 04, 2009 at 03:02:55AM +, Kostas Georgiou wrote: On Sat, Jan 03, 2009 at 06:32:38PM -0600, Matt Domsch wrote: What ext3 journaling options are enabled (e.g. what does 'mount' say)? If it's data=ordered (the default), that's OK. If it's data=journal, then all the data gets

Re: Disk IO issues

2009-01-02 Thread Sascha Thomas Spreitzer
Hello again, this line looks suspicious to me: # nameactive_objs num_objs objsize objperslab pagesperslab : tunables limit batchcount sharedfactor : slabdata active_slabs num_slabs sharedavail ext3_inode_cache 98472 15026076051 : tunables 54 27 8 : slabdata 30052

Re: Disk IO issues

2009-01-02 Thread Michael Schwendt
On Fri, 2 Jan 2009 09:38:43 +0100, Sascha wrote: The Filesystem is btw. marked as needs_recovery. Which can be harmless, because it is a feature flag that is also set if dumpe2fs is run on a mounted fs. It means that there are blocks that still need to be committed, which is pretty normal for a

Re: Disk IO issues

2009-01-02 Thread James Antill
On Fri, 2009-01-02 at 11:57 -0600, Mike McGrath wrote: On Fri, 2 Jan 2009, Sascha Thomas Spreitzer wrote: Hello again, this line looks suspicious to me: # nameactive_objs num_objs objsize objperslab pagesperslab : tunables limit batchcount sharedfactor : slabdata

Re: Disk IO issues

2009-01-02 Thread Mike McGrath
On Fri, 2 Jan 2009, James Antill wrote: On Fri, 2009-01-02 at 11:57 -0600, Mike McGrath wrote: On Fri, 2 Jan 2009, Sascha Thomas Spreitzer wrote: Hello again, this line looks suspicious to me: # nameactive_objs num_objs objsize objperslab pagesperslab : tunables

Re: Disk IO issues

2009-01-02 Thread Mike McGrath
On Fri, 2 Jan 2009, Stephen John Smoogen wrote: On Fri, Jan 2, 2009 at 10:57 AM, Mike McGrath mmcgr...@redhat.com wrote: On Fri, 2 Jan 2009, Sascha Thomas Spreitzer wrote: Hello again, this line looks suspicious to me: # nameactive_objs num_objs objsize objperslab

Re: Disk IO issues

2009-01-02 Thread Stephen John Smoogen
On Fri, Jan 2, 2009 at 12:29 PM, Mike McGrath mmcgr...@redhat.com wrote: On Fri, 2 Jan 2009, Stephen John Smoogen wrote: On Fri, Jan 2, 2009 at 10:57 AM, Mike McGrath mmcgr...@redhat.com wrote: On Fri, 2 Jan 2009, Sascha Thomas Spreitzer wrote: Hello again, this line looks suspicious

Re: Disk IO issues

2009-01-02 Thread Kostas Georgiou
On Fri, Jan 02, 2009 at 01:28:43PM -0600, Mike McGrath wrote: On Fri, 2 Jan 2009, James Antill wrote: On Fri, 2009-01-02 at 11:57 -0600, Mike McGrath wrote: On Fri, 2 Jan 2009, Sascha Thomas Spreitzer wrote: Hello again, this line looks suspicious to me: # name

Re: Disk IO issues

2009-01-02 Thread Jesse Keating
On Fri, 2009-01-02 at 13:28 -0600, Mike McGrath wrote: There are other directories at /mnt/koji/static-repos/. A directory like static-repos contains almost exclusively hardlinks to those packages. Since many of those hardlink oriented directories can be recreated, we don't bother backing

Re: Disk IO issues

2009-01-01 Thread Kostas Georgiou
On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote: On Thu, 1 Jan 2009, James Antill wrote: On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar

Re: Disk IO issues

2009-01-01 Thread Mike McGrath
On Thu, 1 Jan 2009, Kostas Georgiou wrote: On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote: On Thu, 1 Jan 2009, James Antill wrote: On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something.

Re: Disk IO issues

2009-01-01 Thread Stephen John Smoogen
On Thu, Jan 1, 2009 at 12:17 AM, Mike McGrath mmcgr...@redhat.com wrote: On Thu, 1 Jan 2009, James Antill wrote: On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with

Re: Disk IO issues

2009-01-01 Thread Mike McGrath
On Wed, 31 Dec 2008, Sascha Thomas Spreitzer wrote: Hello Mike, maybe the RAID mirror failed and is auto healing in background. Is there a way to determine the RAID state? It dawns on me I never answered these questions. The raid array is fine, its got 14 drives in a raid5 configuration and

Re: Disk IO issues

2009-01-01 Thread Jon Stanley
On Thu, Jan 1, 2009 at 7:17 AM, Kostas Georgiou k.georg...@imperial.ac.uk wrote: Can you run blktrace+seekwatcher (both in EPEL) to get an idea on what is going on? An iostat -x -k /dev/sde 1 output will also be helpfull. Here's a slabinfo that someone else requested and the iostat. I don't

Re: Disk IO issues

2009-01-01 Thread Mike McGrath
On Thu, 1 Jan 2009, Kostas Georgiou wrote: On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote: On Thu, 1 Jan 2009, James Antill wrote: On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something.

Disk IO issues

2008-12-31 Thread Mike McGrath
Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At least I think we are. This is a PERC5/E and MD1000 array. When I try to do a

Re: Disk IO issues

2008-12-31 Thread Xavier Lamien
On Wed, Dec 31, 2008 at 9:42 PM, Mike McGrath mmcgr...@redhat.com wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At

Re: Disk IO issues

2008-12-31 Thread Ricky Zhou
On 2008-12-31 10:49:56 PM, Xavier Lamien wrote: Could you perform an hdparm -tT on that disk ? /dev/sde: Timing cached reads: 2668 MB in 2.00 seconds = 1336.06 MB/sec Timing buffered disk reads: 1024 MB in 3.01 seconds = 340.69 MB/sec Also, output an strace against your cat dd commands.

Re: Disk IO issues

2008-12-31 Thread Corey Chandler
Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At least I think we are. This is a PERC5/E and MD1000

Re: Disk IO issues

2008-12-31 Thread Mike McGrath
On Wed, 31 Dec 2008, Ricky Zhou wrote: On 2008-12-31 10:49:56 PM, Xavier Lamien wrote: Could you perform an hdparm -tT on that disk ? /dev/sde: Timing cached reads: 2668 MB in 2.00 seconds = 1336.06 MB/sec Timing buffered disk reads: 1024 MB in 3.01 seconds = 340.69 MB/sec Also,

Re: Disk IO issues

2008-12-31 Thread Stephen John Smoogen
On Wed, Dec 31, 2008 at 1:42 PM, Mike McGrath mmcgr...@redhat.com wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At

Re: Disk IO issues

2008-12-31 Thread Mike McGrath
On Wed, 31 Dec 2008, Corey Chandler wrote: Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At

Re: Disk IO issues

2008-12-31 Thread Sascha Thomas Spreitzer
If its related to the FS driver ( inode table or algorithms ) the program slabtop might give an indication of the kernel processes eating system performance. Slabtop is in the ps-tools suite, should be on any major linux distribution. 2009/1/1 Greg Swift gregsw...@gmail.com: On Wed, Dec 31, 2008

Re: Disk IO issues

2008-12-31 Thread Mike McGrath
On Thu, 1 Jan 2009, Sascha Thomas Spreitzer wrote: If its related to the FS driver ( inode table or algorithms ) the program slabtop might give an indication of the kernel processes eating system performance. Slabtop is in the ps-tools suite, should be on any major linux distribution.

Re: Disk IO issues

2008-12-31 Thread Mike McGrath
On Wed, 31 Dec 2008, Greg Swift wrote: On Wed, Dec 31, 2008 at 17:35, Mike McGrath mmcgr...@redhat.com wrote: On Wed, 31 Dec 2008, Corey Chandler wrote: Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something.

Re: Disk IO issues

2008-12-31 Thread Mike McGrath
On Wed, 31 Dec 2008, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At least I think we are. This is

Re: Disk IO issues

2008-12-31 Thread James Antill
On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing slower then normal disk IO. At least I think we

Re: Disk IO issues

2008-12-31 Thread Mike McGrath
On Thu, 1 Jan 2009, James Antill wrote: On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote: Lets pool some knowledge together because at this point, I'm missing something. I've been doing all measurements with sar as bonnie, etc, causes builds to timeout. Problem: We're seeing