Re: processes not getting fair share of available disk I/O

2006-12-14 Thread Dieter
Mutex profiling would show if there is a mutex somehow getting in the way of your I/O (e.g. if Giant is somehow being forced). I dont think it would show anything though. You can try to study interrupt issues (e.g. look for an interrupt storm during I/O) with vmstat -i. Other

Re: processes not getting fair share of available disk I/O

2006-12-14 Thread Kris Kennaway
On Thu, Dec 14, 2006 at 09:07:35AM +, Dieter wrote: Mutex profiling would show if there is a mutex somehow getting in the way of your I/O (e.g. if Giant is somehow being forced). I dont think it would show anything though. You can try to study interrupt issues (e.g. look for

Re: processes not getting fair share of available disk I/O

2006-12-14 Thread Dieter
Sorry, yes. Nothing else contended for it though, so it doesn't appear to be a source of performance problems - it is probably a secondary effect from something else. I guess you're running some old version of FreeBSD since those line numbers don't correspond to anything reasonable in the

Re: processes not getting fair share of available disk I/O

2006-12-14 Thread Kris Kennaway
On Thu, Dec 14, 2006 at 02:37:31PM +, Dieter wrote: Sorry, yes. Nothing else contended for it though, so it doesn't appear to be a source of performance problems - it is probably a secondary effect from something else. I guess you're running some old version of FreeBSD since those

Re: processes not getting fair share of available disk I/O

2006-12-14 Thread Dieter
FreeBSD 6.0 Erk. How about retrying with something modern ;-) We do fix lots of bugs over time you know! In my defense, 6.0 is only one revision down. (until 6.2 comes out real soon now) 32.3%Sys 31.5%Intr 0.0%User 0.0%Nice 36.2%Idl 710892 inact32 3: sio1 74.8%Sys

Re: processes not getting fair share of available disk I/O

2006-12-14 Thread Kris Kennaway
On Thu, Dec 14, 2006 at 04:26:18PM +, Dieter wrote: FreeBSD 6.0 Erk. How about retrying with something modern ;-) We do fix lots of bugs over time you know! In my defense, 6.0 is only one revision down. (until 6.2 comes out real soon now) Or to put it another way, your

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-12-13 Thread Kris Kennaway
On Mon, Dec 11, 2006 at 10:32:53AM +, Dieter wrote: Did this problem start before you made port2file run with rtprio? Yes. I only added rtprio because it wasn't working. Can you please include a copy of your kernel configuration file and dmesg? I think you asked that before:

Re: processes not getting fair share of available disk I/O

2006-12-13 Thread Dieter
Is Giant the only mutex/lock that could be a bottleneck across disks? The only one I can think of that is generic. One would have to do more extensive profiling and diagnosis to try and figure out what is wrong with your system. Suggestions of what to look at would be welcome. The only

Re: processes not getting fair share of available disk I/O

2006-12-13 Thread Kris Kennaway
On Wed, Dec 13, 2006 at 04:37:42PM +, Dieter wrote: Is Giant the only mutex/lock that could be a bottleneck across disks? The only one I can think of that is generic. One would have to do more extensive profiling and diagnosis to try and figure out what is wrong with your system.

Re: processes not getting fair share of available disk I/O

2006-12-13 Thread Dieter
Mutex profiling would show if there is a mutex somehow getting in the way of your I/O (e.g. if Giant is somehow being forced). I dont think it would show anything though. You can try to study interrupt issues (e.g. look for an interrupt storm during I/O) with vmstat -i. Other than that

Re: processes not getting fair share of available disk I/O

2006-12-13 Thread Kris Kennaway
On Wed, Dec 13, 2006 at 10:42:03PM +, Dieter wrote: Mutex profiling would show if there is a mutex somehow getting in the way of your I/O (e.g. if Giant is somehow being forced). I dont think it would show anything though. You can try to study interrupt issues (e.g. look for an

processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-12-11 Thread Dieter
Did this problem start before you made port2file run with rtprio? Yes. I only added rtprio because it wasn't working. Can you please include a copy of your kernel configuration file and dmesg? I think you asked that before: :-) OK, that's correct. Can you also provide details of

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-12-08 Thread Kris Kennaway
On Thu, Dec 07, 2006 at 05:41:51PM +, Dieter wrote: However, I don't know what you mean by data is lost. Data should never be lost from the filesystem regardless of how slow the I/O is happening, unless there's something else going wrong (e.g. driver bug). Also, rtprio should

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-12-07 Thread Dieter
hw.ata.wc=3D3D3D0 ^^^ Make my hard drive go rally slow please (just in case I crash)= :) =3D20 Slower, yes, but not *that* slow. =3D20 Normal ls : 0.032 second. Two processes using same disk, multiply by= two, so 0.064 second. Maybe

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-12-07 Thread Kris Kennaway
On Thu, Dec 07, 2006 at 03:21:46PM +, Dieter wrote: hw.ata.wc=3D3D3D0 ^^^ Make my hard drive go rally slow please (just in case I crash)= :) =3D20 Slower, yes, but not *that* slow. =3D20 Normal ls : 0.032 second. Two processes

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-12-07 Thread Dieter
hw.ata.wc=3D3D3D3D0 ^^^ Make my hard drive go rally slow please (just in case I cr= ash)=3D :) =3D3D20 Slower, yes, but not *that* slow. =3D3D20 Normal ls : 0.032 second. Two processes using same disk, multipl= y by=3D

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-23 Thread Dieter
hw.ata.wc=3D3D0 ^^^ Make my hard drive go rally slow please (just in case I crash) :) =20 Slower, yes, but not *that* slow. =20 Normal ls : 0.032 second. Two processes using same disk, multiply by two, so 0.064 second. Maybe the multiplier is more

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-23 Thread Dieter
Here's another oddity: With one process reading from ad4, crunching data, writing to ad2: 4 usersLoad 0.31 0.47 0.67 Nov 23 10:05 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-23 Thread Kris Kennaway
On Thu, Nov 23, 2006 at 09:35:08AM +, Dieter wrote: hw.ata.wc=3D3D0 ^^^ Make my hard drive go rally slow please (just in case I crash) :) =20 Slower, yes, but not *that* slow. =20 Normal ls : 0.032 second. Two processes using same disk,

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Kris Kennaway
On Tue, Nov 21, 2006 at 11:12:38PM +, Dieter wrote: I'm surprised that you're seeing that much of a hang. Even if the disks are busy, the system should slow down all disk processes equally, so no one process blocks, but they're all a little slower. I collected a bit of data: While

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Dieter
In message [EMAIL PROTECTED], Kris Kennaway writes: I'm surprised that you're seeing that much of a hang. Even if the di= sks are busy, the system should slow down all disk processes equally, so no one process blocks, but they're all a little slower. =20 I collected a bit of data:

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Kris Kennaway
On Wed, Nov 22, 2006 at 11:02:54AM +, Dieter wrote: In message [EMAIL PROTECTED], Kris Kennaway writes: I'm surprised that you're seeing that much of a hang. Even if the di= sks are busy, the system should slow down all disk processes equally, so no one process blocks, but

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Dieter
In message [EMAIL PROTECTED], Kris Kennaway writes: --mP3DRpeJDSE+ciuQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 22, 2006 at 11:02:54AM +, Dieter wrote: In message [EMAIL PROTECTED], Kris Kennaway

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Kris Kennaway
On Wed, Nov 22, 2006 at 06:12:06PM +, Dieter wrote: time ls on a small directory on disk2 =3D20 real4m51.911s user0m0.000s sys 0m0.002s =3D20 I expect access to a busy disk to take longer, but 5 minutes is a bit much. And that's the root

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Dieter
time ls on a small directory on disk2 =3D3D20 real4m51.911s user0m0.000s sys 0m0.002s =3D3D20 I expect access to a busy disk to take longer, but 5 minutes is a bit much. And that's the root directory of the filesystem, it didn't have

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Kris Kennaway
On Wed, Nov 22, 2006 at 07:41:46PM +, Dieter wrote: oad have been trimmed from your email. =3D20 In telnet window 1: =3D20 cd /disk1/ cp -ip very_big_file /disk2/bar/ (the workload) =3D20 In telnet window 2: =3D20 time ls /disk3/foo1/

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Dieter
hw.ata.wc=3D0 ^^^ Make my hard drive go rally slow please (just in case I crash) :) Slower, yes, but not *that* slow. Normal ls : 0.032 second. Two processes using same disk, multiply by two, so 0.064 second. Maybe the multiplier is more than 2, call it 10x, so 0.32

Re: processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-22 Thread Kris Kennaway
On Wed, Nov 22, 2006 at 11:29:16PM +, Dieter wrote: hw.ata.wc=3D0 ^^^ Make my hard drive go rally slow please (just in case I crash) :) Slower, yes, but not *that* slow. Normal ls : 0.032 second. Two processes using same disk, multiply by two, so 0.064

processes not getting fair share of available disk I/O (was: Re: TCP parameters and interpreting tcpdump output )

2006-11-21 Thread Dieter
I'm surprised that you're seeing that much of a hang. Even if the disks are busy, the system should slow down all disk processes equally, so no one process blocks, but they're all a little slower. I collected a bit of data: While copying a large file from disk1 to disk2, time ls on a small