Deadlocks (Was: Re: vmstat 'b' (disk busy?) field keeps climbing ...)

2006-06-28 Thread Marc G. Fournier
On Mon, 26 Jun 2006, Kostik Belousov wrote: Core dumps are somewhat unconvenient in this situation. Better, sending report to me, follow my advise in http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html 'k, I'm working on getting a serial console

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-26 Thread Kostik Belousov
On Mon, Jun 26, 2006 at 02:20:12AM -0300, Marc G. Fournier wrote: On Mon, 26 Jun 2006, Kostik Belousov wrote: Yes, this looks like a deadlock. As I understand, that's on 6.1-STABLE ? Yes, kernel sources, it seems, from May 25th, according to my /usr/src tree ... BTW, do you use

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-26 Thread Marc G. Fournier
On Mon, 26 Jun 2006, Kostik Belousov wrote: On Mon, Jun 26, 2006 at 02:20:12AM -0300, Marc G. Fournier wrote: On Mon, 26 Jun 2006, Kostik Belousov wrote: Yes, this looks like a deadlock. As I understand, that's on 6.1-STABLE ? Yes, kernel sources, it seems, from May 25th, according to my

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-26 Thread Marc G. Fournier
I think I might have found *at least* one of the problems, and that being the excessively high blocked states while ps isn't finding anything ... MySQL We just recently started allowing clients to run a MySQL server *within* their vServer ... in a drastic move, I just shut them all down on

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-26 Thread Max Laier
On Monday 26 June 2006 20:25, Marc G. Fournier wrote: I think I might have found *at least* one of the problems, and that being the excessively high blocked states while ps isn't finding anything ... MySQL We just recently started allowing clients to run a MySQL server *within* their

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-26 Thread Marian Hettwer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hej there, Marc G. Fournier wrote: I think I might have found *at least* one of the problems, and that being the excessively high blocked states while ps isn't finding anything ... MySQL We just recently started allowing clients to run a

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-26 Thread Jonathan Noack
Marc G. Fournier wrote: On Mon, 26 Jun 2006, Max Laier wrote: On Monday 26 June 2006 20:25, Marc G. Fournier wrote: I think I might have found *at least* one of the problems, and that being the excessively high blocked states while ps isn't finding anything ... MySQL We just recently

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-25 Thread Dmitry Morozovsky
On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF MGF 'k, stupid question then ... what am I searching for? MGF MGF MGF MGF # ps axlww | awk '{print $9}' | sort | uniq -c | sort -nr MGF MGF Well, try MGF MGF ps axlww | awk '$10 ~ /^D[^L]/' MGF MGF which should give you a list of

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-25 Thread Marc G. Fournier
On Sun, 25 Jun 2006, Dmitry Morozovsky wrote: On Sun, 25 Jun 2006, Marc G. Fournier wrote: MGF Errm... It seems I turn you to the wrong side... Normal disk-locked MGF processes MGF have DL (DL+) state... Well, then try something like MGF MGF ps ax -O ppid,flags,mwchan | awk '($6 ~ /^D/

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-25 Thread Kostik Belousov
On Mon, Jun 26, 2006 at 01:47:04AM -0300, Marc G. Fournier wrote: On Mon, 26 Jun 2006, Marc G. Fournier wrote: 3416 1 1004100 ufs ?? DsJ0:13.01 /usr/local/libexec/postfix/master 3418 3416 1004100 ufs ?? DJ 0:04.16 qmgr -l -t fifo -u 33561 3416 1004100 ufs ?? DJ

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-25 Thread Marc G. Fournier
On Mon, 26 Jun 2006, Marc G. Fournier wrote: 3416 1 1004100 ufs ?? DsJ0:13.01 /usr/local/libexec/postfix/master 3418 3416 1004100 ufs ?? DJ 0:04.16 qmgr -l -t fifo -u 33561 3416 1004100 ufs ?? DJ 0:00.02 smtp -n smtp-amavis -t unix -u -o

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-25 Thread Marc G. Fournier
On Sun, 25 Jun 2006, Marc G. Fournier wrote: # ps ax -O ppid,flags,mwchan | awk '$6 ~ /^D/ || $6 == STAT' PID PPID F MWCHAN TT STAT TIME COMMAND 2 0 204 - ?? DL 0:22.83 [g_event] 3 0 204 - ?? DL 3:13.81 [g_up] 4 0 204 -

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-25 Thread Marc G. Fournier
On Mon, 26 Jun 2006, Kostik Belousov wrote: Yes, this looks like a deadlock. As I understand, that's on 6.1-STABLE ? Yes, kernel sources, it seems, from May 25th, according to my /usr/src tree ... BTW, do you use snapshots ? Not that I've explicitly enabled ... I think that without ddb

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Dmitry Morozovsky
On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF 'b' stands for blocked, not busy. Judging by your page fault rate MGF and the high number of frees and pages being scanned, you're probably MGF swapping tasks in and out and are waiting on disk. Take a look at MGF vmstat -s, and consider

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Kostik Belousov
On Sat, Jun 24, 2006 at 11:55:26AM +0400, Dmitry Morozovsky wrote: On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF 'b' stands for blocked, not busy. Judging by your page fault rate MGF and the high number of frees and pages being scanned, you're probably MGF swapping tasks in and out

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Dmitry Morozovsky
On Sat, 24 Jun 2006, Kostik Belousov wrote: KB MGF 'b' stands for blocked, not busy. Judging by your page fault rate KB MGF and the high number of frees and pages being scanned, you're probably KB MGF swapping tasks in and out and are waiting on disk. Take a look at KB MGF vmstat -s,

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Matthew D. Fuller
On Fri, Jun 23, 2006 at 11:38:44PM -0400 I heard the voice of Chuck Swiger, and lo! it spake thus: Yeah-- it's more common for a system to need more RAM for dynamicly allocated content which would be placed into the swapfile then it uses binary executable pages, it's possible to go the other

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Matthew D. Fuller
On Fri, Jun 23, 2006 at 08:33:05PM -0500 I heard the voice of Matthew D. Fuller, and lo! it spake thus: It's the vnode pager, not the swap pager. AIUI, that's mostly paging in and out pages of running binaries (from the image on disk), not moving stuff in and out of swapspace. Actually, as

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
On Sat, 24 Jun 2006, Matthew D. Fuller wrote: On Fri, Jun 23, 2006 at 11:38:44PM -0400 I heard the voice of Chuck Swiger, and lo! it spake thus: Yeah-- it's more common for a system to need more RAM for dynamicly allocated content which would be placed into the swapfile then it uses binary

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
On Sat, 24 Jun 2006, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 11:55:26AM +0400, Dmitry Morozovsky wrote: On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF 'b' stands for blocked, not busy. Judging by your page fault rate MGF and the high number of frees and pages being scanned, you're

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
On Sat, 24 Jun 2006, Dmitry Morozovsky wrote: On Sat, 24 Jun 2006, Kostik Belousov wrote: KB MGF 'b' stands for blocked, not busy. Judging by your page fault rate KB MGF and the high number of frees and pages being scanned, you're probably KB MGF swapping tasks in and out and are

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Dmitry Morozovsky
On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF MGF is there a way of finding out what processes are blocked? MGF MGF Aren't they in 'D' status by ps? MGF Use ps axlww. In this way, at least actual blocking points are shown. MGF MGF 'k, stupid question then ... what am I searching for?

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
On Sat, 24 Jun 2006, Dmitry Morozovsky wrote: On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF MGF is there a way of finding out what processes are blocked? MGF MGF Aren't they in 'D' status by ps? MGF Use ps axlww. In this way, at least actual blocking points are shown. MGF MGF 'k,

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Kostik Belousov
On Sat, Jun 24, 2006 at 02:57:27PM -0300, Marc G. Fournier wrote: On Sat, 24 Jun 2006, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 11:55:26AM +0400, Dmitry Morozovsky wrote: On Sat, 24 Jun 2006, Marc G. Fournier wrote: MGF 'b' stands for blocked, not busy. Judging by your page fault

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Kostik Belousov
On Sat, Jun 24, 2006 at 09:52:03PM +0300, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 02:57:27PM -0300, Marc G. Fournier wrote: On Sat, 24 Jun 2006, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 11:55:26AM +0400, Dmitry Morozovsky wrote: On Sat, 24 Jun 2006, Marc G. Fournier wrote:

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
On Sat, 24 Jun 2006, Kostik Belousov wrote: 2. 2 Giant holders/lockers. Is it constant ? Are the processes holding/waiting for Giant are the same ? Mostly appears to be 'clock' ... pluto# ps axlww | grep Giant | grep -v grep 012 0 0 -32 0 0 8 Giant LL??3:07.03

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
On Sat, 24 Jun 2006, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 09:52:03PM +0300, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 02:57:27PM -0300, Marc G. Fournier wrote: On Sat, 24 Jun 2006, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 11:55:26AM +0400, Dmitry Morozovsky wrote: On

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Kostik Belousov
On Sat, Jun 24, 2006 at 04:45:49PM -0300, Marc G. Fournier wrote: On Sat, 24 Jun 2006, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 09:52:03PM +0300, Kostik Belousov wrote: On Sat, Jun 24, 2006 at 02:57:27PM -0300, Marc G. Fournier wrote: On Sat, 24 Jun 2006, Kostik Belousov wrote: On

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-24 Thread Marc G. Fournier
'k, this has gotta be a leak somewhere ... I'm now up to 6 blocked: 0 8 0 7449224 236552 213 2 1 0 109 0 101 0 475 2890 2143 2 6 92 0 6 0 7481104 247704 578 0 0 0 1196 0 262 0 808 8901 3049 5 16 79 0 6 0 7450820 253576 1385 3 2 3 1379 0 13 0 303 4742

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Charles Swiger
On Jun 23, 2006, at 4:44 PM, Marc G. Fournier wrote: procs memory pagedisks faults cpu r b w avmfre flt re pi po fr sr da0 pa0 in sy cs us sy id 1 42 1 10249060 161668 1290 54 12 3 1409 2202 102 0 751 6416 3350 24 15 61 0

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Marc G. Fournier
On Fri, 23 Jun 2006, Charles Swiger wrote: On Jun 23, 2006, at 4:44 PM, Marc G. Fournier wrote: procs memory pagedisks faults cpu r b w avmfre flt re pi po fr sr da0 pa0 in sy cs us sy id 1 42 1 10249060 161668 1290 54 12 3 1409

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Matthew D. Fuller
On Fri, Jun 23, 2006 at 07:12:54PM -0300 I heard the voice of Marc G. Fournier, and lo! it spake thus: 31750 vnode pager pageins 209538 vnode pager pages paged in 15954 vnode pager pageouts 219494 vnode pager pages paged out This may be something to look at. My workstation

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Marc G. Fournier
On Fri, 23 Jun 2006, Matthew D. Fuller wrote: On Fri, Jun 23, 2006 at 07:12:54PM -0300 I heard the voice of Marc G. Fournier, and lo! it spake thus: 31750 vnode pager pageins 209538 vnode pager pages paged in 15954 vnode pager pageouts 219494 vnode pager pages paged out This

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Matthew D. Fuller
On Fri, Jun 23, 2006 at 10:02:22PM -0300 I heard the voice of Marc G. Fournier, and lo! it spake thus: Which is odd, no, if I'm hardly swapping? Well, 31750 vnode pager pageins 15954 vnode pager pageouts It's the vnode pager, not the swap pager. AIUI, that's mostly paging in and

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Marc G. Fournier
On Fri, 23 Jun 2006, Matthew D. Fuller wrote: On Fri, Jun 23, 2006 at 10:02:22PM -0300 I heard the voice of Marc G. Fournier, and lo! it spake thus: Which is odd, no, if I'm hardly swapping? Well, 31750 vnode pager pageins 15954 vnode pager pageouts It's the vnode pager, not the

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Mark Linimon
On Fri, Jun 23, 2006 at 11:02:35PM -0300, Marc G. Fournier wrote: Just bothers me that its been running 4.x for 3 years now, no problems and as soon as I upgrade it to 6.x, all the headaches start :( I am well familiar with Mr. Murphy and all his works :-) mcl

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Chuck Swiger
Marc G. Fournier wrote: [ ... ] 31750 vnode pager pageins 15954 vnode pager pageouts It's the vnode pager, not the swap pager. AIUI, that's mostly paging in and out pages of running binaries (from the image on disk), not moving stuff in and out of swapspace. ah, okay ... Yeah-- it's

Re: vmstat 'b' (disk busy?) field keeps climbing ...

2006-06-23 Thread Marc G. Fournier
On Fri, 23 Jun 2006, Charles Swiger wrote: On Jun 23, 2006, at 4:44 PM, Marc G. Fournier wrote: procs memory pagedisks faults cpu r b w avmfre flt re pi po fr sr da0 pa0 in sy cs us sy id 1 42 1 10249060 161668 1290 54 12 3 1409