Re: Lock order reversals in sys_pipe.c and kern_sig.c
[2002-11-18 11:39] Alfred Perlstein said:
| * Kris Kennaway [EMAIL PROTECTED] [021118 11:06] wrote:
| I've just turned witness back on on the bento cluster, and got the
| following lock order reversals a number of times overnight:
|
| Nov 18 07:45:40 user.crit gohan11 kernel: 1st 0xc6887200 pipe mutex (pipe mutex)
|@ /local0/src-client/sys/kern/sys_pipe.c:465
| Nov 18 07:45:40 user.crit gohan11 kernel: 2nd 0xc0447780 sigio lock (sigio lock)
|@ /local0/src-client/sys/kern/kern_sig.c:2225
| Nov 18 10:28:47 user.crit gohan10 kernel: 1st 0xc4941580 pipe mutex (pipe mutex)
|@ /local0/src-client/sys/kern/sys_pipe.c:1038
| [...]
|
| Are these known problems?
|
| Well now they are, I will investigate as time permits.
Maybe this will help. The attached prog causes a LOR message. I dug
thru the kernel source from fcntl to fsetown, but am little more than
lost at this point...
thanks.
brent
--
Develop your talent, man, and leave the world something. Records are
really gifts from people. To think that an artist would love you enough
to share his music with anyone is a beautiful thing. -- Duane Allman
#include unistd.h
#include fcntl.h
#include stdio.h
/*
This program causes the following LOR. Strangely, it will only cause
the LOR _once_. To repeat, you must reboot.
lock order reversal
1st 0xc526c180 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465
2nd 0xc051ef80 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225
Debugger(witness_lock)
Stopped at Debugger+0x54: xchgl %ebx,in_Debugger.0
db trace
Debugger(c04961d5,c051ef80,c04ccf5a,c04ccf5a,c04cf7e1) at Debugger+0x54
witness_lock(c051ef80,8,c04cf7e1,8b1,c526c180) at witness_lock+0x667
_mtx_lock_flags(c051ef80,0,c04cf7e1,8b1,23) at _mtx_lock_flags+0xb1
pgsigio(c4f2fe58,17,0,1ad,0) at pgsigio+0x30
pipe_read(c4f28654,e0286c7c,c5024680,0,c4de28c0) at pipe_read+0x516
dofileread(c4de28c0,c4f28654,3,bfbffca3,1) at dofileread+0xd2
read(c4de28c0,e0286d10,c04e9042,407,3) at read+0x6b
syscall(2f,2f,2f,bfbffcd8,bfbffce0) at syscall+0x28e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (3, FreeBSD ELF32, read), eip = 0x280b09c3, esp = 0xbfbffc7c, ebp =
0xbfbffcb0 ---
*/
int
main(int argc, char** argv)
{
int fd[2];
long flags;
char buf[1];
if (pipe(fd) == 0) {
flags = fcntl(fd[0], F_GETFL, 0);
fcntl(fd[0], F_SETFL, flags | O_NONBLOCK | O_ASYNC);
fcntl(fd[0], F_SETOWN, getpid());/* -- causes LOR in read() */
read(fd[0], buf, 1);
close(fd[0]);
close(fd[1]);
}
else
{
perror(fifo);
}
return 0;
}
Re: Lock order reversals in sys_pipe.c and kern_sig.c
[2002-12-04 15:12] Alfred Perlstein said: | * Lars Eggert [EMAIL PROTECTED] [021204 15:04] wrote: | | FYI, just got a similar one yesterday on an up-to-date -current: | | lock order reversal | 1st 0xc784d700 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 | 2nd 0xc0513840 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 | | Since no one has given me the traceback based on the patch I posted | earlier I have ceased to care about the problem. If someone | seriously cares they will run with my patch and post a backtrace. Running java _always_ causes one lock order warning -- and only one, AFAICT, after having the system up for a few days; successive execs of the java binary do not cause the warning. thanks brent brent@scratch$ uname -a FreeBSD scratch.rcfile.org 5.0-CURRENT FreeBSD 5.0-CURRENT #15: Mon Dec 16 23:22:55 EST 2002 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SCRATCH i386 brent@scratch$ javavm -version lock order reversal 1st 0xc504b280 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 2nd 0xc051d660 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 Debugger(witness_lock) Stopped at Debugger+0x54: xchgl %ebx,in_Debugger.0 db trace Debugger(c0494b35,c051d660,c04cb89d,c04cb89d,c04ce124) at Debugger+0x54 witness_lock(c051d660,8,c04ce124,8b1,c504b280) at witness_lock+0x667 _mtx_lock_flags(c051d660,0,c04ce124,8b1,c4f29d10) at _mtx_lock_flags+0xb1 pgsigio(c4f29da8,17,0,1ad,c0598898) at pgsigio+0x30 pipe_read(c4f21690,e0297c7c,c52a7200,0,c4de0c40) at pipe_read+0x596 dofileread(c4de0c40,c4f21690,3,bfbff0a7,1) at dofileread+0xd2 read(c4de0c40,e0297d10,c04e77be,407,3) at read+0x6b syscall(2f,2f,2f,281d9c02,bfbff0c0) at syscall+0x28e Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (3, FreeBSD ELF32, read), eip = 0x280c6993, esp = 0xbfbff08c, ebp = 0xbfbff0e8 --- db continue java version 1.3.1-p7 Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1-p7-root-021107-04:35) Classic VM (build 1.3.1-p7-root-021107-04:35, green threads, nojit) brent@scratch$ -- Develop your talent, man, and leave the world something. Records are really gifts from people. To think that an artist would love you enough to share his music with anyone is a beautiful thing. -- Duane Allman To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
On 04-Dec-2002 Alfred Perlstein wrote:
* Lars Eggert [EMAIL PROTECTED] [021204 15:04] wrote:
FYI, just got a similar one yesterday on an up-to-date -current:
lock order reversal
1st 0xc784d700 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465
2nd 0xc0513840 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225
Since no one has given me the traceback based on the patch I posted
earlier I have ceased to care about the problem. If someone
seriously cares they will run with my patch and post a backtrace.
Your patch doesn't work because it makes a bogus assumption. A better
try would be this:
Index: subr_witness.c
===
RCS file: /usr/cvs/src/sys/kern/subr_witness.c,v
retrieving revision 1.130
diff -u -r1.130 subr_witness.c
--- subr_witness.c 11 Nov 2002 16:36:20 - 1.130
+++ subr_witness.c 18 Nov 2002 20:42:35 -
@@ -205,6 +205,9 @@
{ uidinfo hash, lock_class_mtx_sleep },
{ uidinfo struct, lock_class_mtx_sleep },
{ NULL, NULL },
+ { sigio lock, lock_class_mtx_sleep },
+ { pipe mutex, lock_class_mtx_sleep },
+ { NULL, NULL },
/*
* spin locks
*/
--
John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/
Power Users Use the Power to Serve! - http://www.FreeBSD.org/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
Alfred Perlstein wrote: * Kris Kennaway [021118 11:06] wrote: I've just turned witness back on on the bento cluster, and got the following lock order reversals a number of times overnight: Nov 18 07:45:40 gohan11 kernel: 1st 0xc6887200 pipe mutex (pipe mutex) @ /local0/src-client/sys/kern/sys_pipe.c:465 Nov 18 07:45:40 gohan11 kernel: 2nd 0xc0447780 sigio lock (sigio lock) @ /local0/src-client/sys/kern/kern_sig.c:2225 Nov 18 10:28:47 gohan10 kernel: 1st 0xc4941580 pipe mutex (pipe mutex) @ /local0/src-client/sys/kern/sys_pipe.c:1038 [...] Are these known problems? Well now they are, I will investigate as time permits. FYI, just got a similar one yesterday on an up-to-date -current: lock order reversal 1st 0xc784d700 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 2nd 0xc0513840 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 Lars -- Lars Eggert [EMAIL PROTECTED] USC Information Sciences Institute smime.p7s Description: S/MIME Cryptographic Signature
Re: Lock order reversals in sys_pipe.c and kern_sig.c
* Lars Eggert [EMAIL PROTECTED] [021204 15:04] wrote: FYI, just got a similar one yesterday on an up-to-date -current: lock order reversal 1st 0xc784d700 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 2nd 0xc0513840 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 Since no one has given me the traceback based on the patch I posted earlier I have ceased to care about the problem. If someone seriously cares they will run with my patch and post a backtrace. thank you, -Alfred To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
Alfred Perlstein wrote: * Lars Eggert [021204 15:04] wrote: FYI, just got a similar one yesterday on an up-to-date -current: lock order reversal 1st 0xc784d700 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 2nd 0xc0513840 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 Since no one has given me the traceback based on the patch I posted earlier I have ceased to care about the problem. If someone seriously cares they will run with my patch and post a backtrace. Applied your patch, will send a traceback when it happens again (I only get one of these every few days). Lars -- Lars Eggert [EMAIL PROTECTED] USC Information Sciences Institute smime.p7s Description: S/MIME Cryptographic Signature
Re: Lock order reversals in sys_pipe.c and kern_sig.c
Alfred Perlstein wrote: * Lars Eggert [021204 15:04] wrote: FYI, just got a similar one yesterday on an up-to-date -current: lock order reversal 1st 0xc784d700 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 2nd 0xc0513840 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 Since no one has given me the traceback based on the patch I posted earlier I have ceased to care about the problem. If someone seriously cares they will run with my patch and post a backtrace. Here's one: lock order reversal 1st 0xc795ea00 pipe mutex (pipe mutex) @ /usr/src/sys/kern/sys_pipe.c:465 2nd 0xc0515340 sigio lock (sigio lock) @ /usr/src/sys/kern/kern_sig.c:2225 Debugger(witness_lock) Stopped at Debugger+0x5a: xchgl %ebx,in_Debugger.0 db trace Debugger(c0472495,c0515340,c04a8e51,c04a8e51,c04ab721) at Debugger+0x5a witness_lock(c0515340,8,c04ab721,8b1,eb4d8be8) at witness_lock+0x142 _mtx_lock_flags(c0515340,0,c04ab721,8b1,c02f1531) at _mtx_lock_flags+0x63 pgsigio(c77afae8,17,0,1ad,0) at pgsigio+0x2d pipe_read(c6ff8384,eb4d8c78,c6beb100,0,c6ac9e00) at pipe_read+0x2d4 dofileread(c6ac9e00,c6ff8384,3,bfbfe317,1) at dofileread+0xd7 read(c6ac9e00,eb4d8d10,c04c9811,407,c6bced50) at read+0x7b syscall(2f,2f,bfbf002f,281dca2b,bfbfe330) at syscall+0x3c6 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (3, FreeBSD ELF32, read), eip = 0x280c5ff3, esp = 0xbfbfe2fc, ebp = 0xbfbfe358 --- -- Lars Eggert [EMAIL PROTECTED] USC Information Sciences Institute smime.p7s Description: S/MIME Cryptographic Signature
Lock order reversals in sys_pipe.c and kern_sig.c
I've just turned witness back on on the bento cluster, and got the following lock order reversals a number of times overnight: Nov 18 07:45:40 user.crit gohan11 kernel: 1st 0xc6887200 pipe mutex (pipe mutex) @ /local0/src-client/sys/kern/sys_pipe.c:465 Nov 18 07:45:40 user.crit gohan11 kernel: 2nd 0xc0447780 sigio lock (sigio lock) @ /local0/src-client/sys/kern/kern_sig.c:2225 Nov 18 10:28:47 user.crit gohan10 kernel: 1st 0xc4941580 pipe mutex (pipe mutex) @ /local0/src-client/sys/kern/sys_pipe.c:1038 [...] Are these known problems? Kris msg46871/pgp0.pgp Description: PGP signature
Re: Lock order reversals in sys_pipe.c and kern_sig.c
* Kris Kennaway [EMAIL PROTECTED] [021118 11:06] wrote: I've just turned witness back on on the bento cluster, and got the following lock order reversals a number of times overnight: Nov 18 07:45:40 user.crit gohan11 kernel: 1st 0xc6887200 pipe mutex (pipe mutex) @ /local0/src-client/sys/kern/sys_pipe.c:465 Nov 18 07:45:40 user.crit gohan11 kernel: 2nd 0xc0447780 sigio lock (sigio lock) @ /local0/src-client/sys/kern/kern_sig.c:2225 Nov 18 10:28:47 user.crit gohan10 kernel: 1st 0xc4941580 pipe mutex (pipe mutex) @ /local0/src-client/sys/kern/sys_pipe.c:1038 [...] Are these known problems? Well now they are, I will investigate as time permits. I'm still in a holding pattern about adding more debugging info to lockd now that we're in release candidate mode. I may ask for a branch to be done, but i'm not sure yet. thank you, -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
On Mon, Nov 18, 2002 at 11:39:01AM -0800, Alfred Perlstein wrote: Well now they are, I will investigate as time permits. Thanks. I'm still in a holding pattern about adding more debugging info to lockd now that we're in release candidate mode. I may ask for a branch to be done, but i'm not sure yet. Feel free to give me non-committed patches to test :-) Kris msg46878/pgp0.pgp Description: PGP signature
Re: Lock order reversals in sys_pipe.c and kern_sig.c
* Kris Kennaway [EMAIL PROTECTED] [021118 12:01] wrote:
On Mon, Nov 18, 2002 at 11:39:01AM -0800, Alfred Perlstein wrote:
Well now they are, I will investigate as time permits.
Thanks.
I'm still in a holding pattern about adding more debugging info to
lockd now that we're in release candidate mode. I may ask for a
branch to be done, but i'm not sure yet.
Feel free to give me non-committed patches to test :-)
GRR, witless always gives me the locations of the correct lock
order, never the locations of the incorrect ones, so here, try this
and give me a backtrace, it should nail you with extreme prejudice
when you get what i think is the wrong order:
Index: pipe.h
===
RCS file: /home/ncvs/src/sys/sys/pipe.h,v
retrieving revision 1.20
diff -u -r1.20 pipe.h
--- pipe.h 14 Aug 2002 01:04:43 - 1.20
+++ pipe.h 18 Nov 2002 20:13:13 -
@@ -112,7 +112,11 @@
};
#define PIPE_MTX(pipe) (pipe)-pipe_mtxp
-#define PIPE_LOCK(pipe)mtx_lock(PIPE_MTX(pipe))
+#define PIPE_LOCK(pipe)\
+ do {\
+ mtx_lock(PIPE_MTX(pipe)); \
+ mtx_assert(sigio_lock, MA_NOTOWNED); \
+ } while (0)
#define PIPE_UNLOCK(pipe) mtx_unlock(PIPE_MTX(pipe))
#define PIPE_LOCK_ASSERT(pipe, type) mtx_assert(PIPE_MTX(pipe), (type))
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
On 18-Nov-2002 Alfred Perlstein wrote:
* Kris Kennaway [EMAIL PROTECTED] [021118 12:01] wrote:
On Mon, Nov 18, 2002 at 11:39:01AM -0800, Alfred Perlstein wrote:
Well now they are, I will investigate as time permits.
Thanks.
I'm still in a holding pattern about adding more debugging info to
lockd now that we're in release candidate mode. I may ask for a
branch to be done, but i'm not sure yet.
Feel free to give me non-committed patches to test :-)
GRR, witless always gives me the locations of the correct lock
order, never the locations of the incorrect ones, so here, try this
and give me a backtrace, it should nail you with extreme prejudice
when you get what i think is the wrong order:
This is why there is a static lock-order list that you can add to.
For example:
Index: subr_witness.c
===
RCS file: /usr/cvs/src/sys/kern/subr_witness.c,v
retrieving revision 1.130
diff -u -r1.130 subr_witness.c
--- subr_witness.c 11 Nov 2002 16:36:20 - 1.130
+++ subr_witness.c 18 Nov 2002 20:42:35 -
@@ -205,6 +205,9 @@
{ uidinfo hash, lock_class_mtx_sleep },
{ uidinfo struct, lock_class_mtx_sleep },
{ NULL, NULL },
+ { sigio lock, lock_class_mtx_sleep },
+ { pipe mutex, lock_class_mtx_sleep },
+ { NULL, NULL },
/*
* spin locks
*/
--
John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/
Power Users Use the Power to Serve! - http://www.FreeBSD.org/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
On 18-Nov-2002 John Baldwin wrote:
On 18-Nov-2002 Alfred Perlstein wrote:
* Kris Kennaway [EMAIL PROTECTED] [021118 12:01] wrote:
On Mon, Nov 18, 2002 at 11:39:01AM -0800, Alfred Perlstein wrote:
Well now they are, I will investigate as time permits.
Thanks.
I'm still in a holding pattern about adding more debugging info to
lockd now that we're in release candidate mode. I may ask for a
branch to be done, but i'm not sure yet.
Feel free to give me non-committed patches to test :-)
GRR, witless always gives me the locations of the correct lock
order, never the locations of the incorrect ones, so here, try this
and give me a backtrace, it should nail you with extreme prejudice
when you get what i think is the wrong order:
This is why there is a static lock-order list that you can add to.
For example:
Index: subr_witness.c
===
RCS file: /usr/cvs/src/sys/kern/subr_witness.c,v
retrieving revision 1.130
diff -u -r1.130 subr_witness.c
--- subr_witness.c 11 Nov 2002 16:36:20 - 1.130
+++ subr_witness.c 18 Nov 2002 20:42:35 -
@@ -205,6 +205,9 @@
{ uidinfo hash, lock_class_mtx_sleep },
{ uidinfo struct, lock_class_mtx_sleep },
{ NULL, NULL },
+ { sigio lock, lock_class_mtx_sleep },
+ { pipe mutex, lock_class_mtx_sleep },
+ { NULL, NULL },
/*
* spin locks
*/
At some point I may add a userland function that allows you to specify
an order to witness at runtime.
--
John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/
Power Users Use the Power to Serve! - http://www.FreeBSD.org/
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c and kern_sig.c
On Mon, Nov 18, 2002 at 12:18:08PM -0800, Alfred Perlstein wrote: * Kris Kennaway [EMAIL PROTECTED] [021118 12:01] wrote: On Mon, Nov 18, 2002 at 11:39:01AM -0800, Alfred Perlstein wrote: Well now they are, I will investigate as time permits. Thanks. I'm still in a holding pattern about adding more debugging info to lockd now that we're in release candidate mode. I may ask for a branch to be done, but i'm not sure yet. Feel free to give me non-committed patches to test :-) GRR, witless always gives me the locations of the correct lock order, never the locations of the incorrect ones, so here, try this and give me a backtrace, it should nail you with extreme prejudice when you get what i think is the wrong order: Thanks, I'll try this ASAP. Kris msg46885/pgp0.pgp Description: PGP signature
Re: Lock order reversals in sys_pipe.c
On 26-Mar-2002 Alfred Perlstein wrote: * Alfred Perlstein [EMAIL PROTECTED] [020326 14:43] wrote: * Kris Kennaway [EMAIL PROTECTED] [020324 14:26] wrote: The bento cluster is now running with WITNESS enabled to try and track down some odd UMA lock corruption panics. Instead, it found the following lock order reversal in sys_pipe.c overnight: Mar 24 09:02:29 user.crit gohan13 kernel: lock order reversal Mar 24 09:02:29 user.crit gohan13 kernel: 1st 0xd99d6500 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 09:02:29 user.crit gohan13 kernel: 2nd 0xd971cddc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Those source references are from a -current kernel from last night. Are you %100 on that? How did you get this to happen? I can see where I hold the pipe lock, then try to get a proc lock, but not the other way around... Any ideas? Note that I can pretty trivially fix this by dropping the pipe lock while calling pgsigio in pipeselwakeup(), however I will then have to make sure the callers of pipeselwakeup() can deal with someone else mucking with the pipe during the call. Anyhow, wtf doesn't witness print out at least one instance of the old locking? Basically let me know where locking is happening in the other direction? That's easy to achieve. Add a static lock ordering in the compiled in arrays, then it will let you know when you do the other way. It may be an indirect relationship and not a direct one as well. The allproc/filedesc thing was an instance of indirect relationships determining the order. -- John Baldwin [EMAIL PROTECTED]http://www.FreeBSD.org/~jhb/ Power Users Use the Power to Serve! - http://www.FreeBSD.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c
* Kris Kennaway [EMAIL PROTECTED] [020324 14:26] wrote: The bento cluster is now running with WITNESS enabled to try and track down some odd UMA lock corruption panics. Instead, it found the following lock order reversal in sys_pipe.c overnight: Mar 24 07:31:44 user.crit gohan17 kernel: lock order reversal Mar 24 07:31:44 user.crit gohan17 kernel: 1st 0xcf51aa80 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:31:44 user.crit gohan17 kernel: 2nd 0xcf88dadc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:12 user.crit gohan10 kernel: lock order reversal Mar 24 07:32:12 user.crit gohan10 kernel: 1st 0xd9a29dc0 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:12 user.crit gohan10 kernel: 2nd 0xd961addc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:57 user.crit gohan12 kernel: lock order reversal Mar 24 07:32:57 user.crit gohan12 kernel: 1st 0xd9423080 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:57 user.crit gohan12 kernel: 2nd 0xdaa704dc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 09:02:29 user.crit gohan13 kernel: lock order reversal Mar 24 09:02:29 user.crit gohan13 kernel: 1st 0xd99d6500 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 09:02:29 user.crit gohan13 kernel: 2nd 0xd971cddc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Those source references are from a -current kernel from last night. Are you %100 on that? How did you get this to happen? I can see where I hold the pipe lock, then try to get a proc lock, but not the other way around... Any ideas? -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' Tax deductible donations for FreeBSD: http://www.freebsdfoundation.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c
* Alfred Perlstein [EMAIL PROTECTED] [020326 14:43] wrote: * Kris Kennaway [EMAIL PROTECTED] [020324 14:26] wrote: The bento cluster is now running with WITNESS enabled to try and track down some odd UMA lock corruption panics. Instead, it found the following lock order reversal in sys_pipe.c overnight: Mar 24 09:02:29 user.crit gohan13 kernel: lock order reversal Mar 24 09:02:29 user.crit gohan13 kernel: 1st 0xd99d6500 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 09:02:29 user.crit gohan13 kernel: 2nd 0xd971cddc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Those source references are from a -current kernel from last night. Are you %100 on that? How did you get this to happen? I can see where I hold the pipe lock, then try to get a proc lock, but not the other way around... Any ideas? Note that I can pretty trivially fix this by dropping the pipe lock while calling pgsigio in pipeselwakeup(), however I will then have to make sure the callers of pipeselwakeup() can deal with someone else mucking with the pipe during the call. Anyhow, wtf doesn't witness print out at least one instance of the old locking? Basically let me know where locking is happening in the other direction? -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' Tax deductible donations for FreeBSD: http://www.freebsdfoundation.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: Lock order reversals in sys_pipe.c
On Tue, Mar 26, 2002 at 02:43:07PM -0800, Alfred Perlstein wrote: * Kris Kennaway [EMAIL PROTECTED] [020324 14:26] wrote: The bento cluster is now running with WITNESS enabled to try and track down some odd UMA lock corruption panics. Instead, it found the following lock order reversal in sys_pipe.c overnight: Mar 24 07:31:44 user.crit gohan17 kernel: lock order reversal Mar 24 07:31:44 user.crit gohan17 kernel: 1st 0xcf51aa80 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:31:44 user.crit gohan17 kernel: 2nd 0xcf88dadc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:12 user.crit gohan10 kernel: lock order reversal Mar 24 07:32:12 user.crit gohan10 kernel: 1st 0xd9a29dc0 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:12 user.crit gohan10 kernel: 2nd 0xd961addc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:57 user.crit gohan12 kernel: lock order reversal Mar 24 07:32:57 user.crit gohan12 kernel: 1st 0xd9423080 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:57 user.crit gohan12 kernel: 2nd 0xdaa704dc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 09:02:29 user.crit gohan13 kernel: lock order reversal Mar 24 09:02:29 user.crit gohan13 kernel: 1st 0xd99d6500 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 09:02:29 user.crit gohan13 kernel: 2nd 0xd971cddc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Those source references are from a -current kernel from last night. Are you %100 on that? How did you get this to happen? Yes. I wasn't doing anything special, unless you count building 5-10 packages simultaneously for a period of 24 hours counts as special ;-) I had to increase the witness limits in order to stop getting the following: witness_get: witness exhausted Well, I eventually got it anyway on all of the machines, but it didn't happen instantly like it did with the default values ;-) I haven't seen any more witness warnings, but it's probably only because it ran out of resources. I expect I'll see it again when I reboot the cluster. Kris msg36603/pgp0.pgp Description: PGP signature
Lock order reversals in sys_pipe.c
The bento cluster is now running with WITNESS enabled to try and track down some odd UMA lock corruption panics. Instead, it found the following lock order reversal in sys_pipe.c overnight: Mar 24 07:31:44 user.crit gohan17 kernel: lock order reversal Mar 24 07:31:44 user.crit gohan17 kernel: 1st 0xcf51aa80 pipe mutex /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:31:44 user.crit gohan17 kernel: 2nd 0xcf88dadc process lock /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:12 user.crit gohan10 kernel: lock order reversal Mar 24 07:32:12 user.crit gohan10 kernel: 1st 0xd9a29dc0 pipe mutex /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:12 user.crit gohan10 kernel: 2nd 0xd961addc process lock /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:57 user.crit gohan12 kernel: lock order reversal Mar 24 07:32:57 user.crit gohan12 kernel: 1st 0xd9423080 pipe mutex /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:57 user.crit gohan12 kernel: 2nd 0xdaa704dc process lock /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 09:02:29 user.crit gohan13 kernel: lock order reversal Mar 24 09:02:29 user.crit gohan13 kernel: 1st 0xd99d6500 pipe mutex /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 09:02:29 user.crit gohan13 kernel: 2nd 0xd971cddc process lock /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Those source references are from a -current kernel from last night. Kris msg36525/pgp0.pgp Description: PGP signature
Re: Lock order reversals in sys_pipe.c
On Sunday 24 March 2002 05:26 pm, you wrote: The bento cluster is now running with WITNESS enabled to try and track down some odd UMA lock corruption panics. Instead, it found the following lock order reversal in sys_pipe.c overnight: Mar 24 07:31:44 user.crit gohan17 kernel: lock order reversal Mar 24 07:31:44 user.crit gohan17 kernel: 1st 0xcf51aa80 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:31:44 user.crit gohan17 kernel: 2nd 0xcf88dadc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:12 user.crit gohan10 kernel: lock order reversal Mar 24 07:32:12 user.crit gohan10 kernel: 1st 0xd9a29dc0 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:12 user.crit gohan10 kernel: 2nd 0xd961addc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 07:32:57 user.crit gohan12 kernel: lock order reversal Mar 24 07:32:57 user.crit gohan12 kernel: 1st 0xd9423080 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 07:32:57 user.crit gohan12 kernel: 2nd 0xdaa704dc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Mar 24 09:02:29 user.crit gohan13 kernel: lock order reversal Mar 24 09:02:29 user.crit gohan13 kernel: 1st 0xd99d6500 pipe mutex @ /local0/scratch/usr/src/sys/kern/sys_pipe.c:450 Mar 24 09:02:29 user.crit gohan13 kernel: 2nd 0xd971cddc process lock @ /local0/scratch/usr/src/sys/kern/kern_sig.c:2093 Those source references are from a -current kernel from last night. Kris I think I found something similar as well, I also found a panic, but havn't had time to track it down. The panic doesn't occur with a kernel from march 13th, but occurs with the latest kernels. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
