Re: panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
On Tue, Jan 28, 2014 at 09:02:31PM -0800, Philip Guenther wrote: On Tue, Jan 28, 2014 at 4:55 PM, patrick keshishian sids...@boxsoft.com wrote: Side note 2: I see gdb crash quite often on subsequent run of app being debugged. I've noticed this on a bunch of snapshots dating some months back. Never reported it cause, I don't have a simple test-case to submit, and things have been moving fast with what you guys are doing. gdb has had many bugs in the past. Many are fixed in versions newer than the version that are shipped with OpenBSD, but licensing keeps us from including them in general. :-/ Anyone care to hack on gdb (no copying GPLv3 source!)? Note that a recent gdb is available in ports, devel/gdb which installs the binary as egdb. Maybe it doesnt know/have tweaks for all our specificities, but it's there. For having tried it on firefox, it didnt fare much better than the base one.. Landry
Re: panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
On Wed, Jan 29, 2014 at 07:50:34AM +0100, J??r??mie Courr??ges-Anglas wrote: patrick keshishian sids...@boxsoft.com writes: On Tue, Jan 28, 2014 at 09:02:31PM -0800, Philip Guenther wrote: On Tue, Jan 28, 2014 at 4:55 PM, patrick keshishian sids...@boxsoft.com wrote: Side note 1: Incidentally, I couldn't figure out how to correctly leave a symbol unresolved while compiling myapp, until dlopen() time. If you can include the shared-object/library in the link command line, then the symbol can be referenced at link time. That lets you refer to the symbols directly as if they were in the executable itself, though you must still declare them for the compiler to know what type they are, etc. A bit of a chicken-and-egg situation for this case. Unless I do some ugly hopping around in Makefiles, it would be difficult to include the object on the link line for the main program. If you need to delay the loading of the shared-object until after process startup via dlopen(), then the executable should use dlsym() to get the symbol address, passing it the handle that dlopen() returned. Yes. I do that right now with a bit of dance. However, reading a bit about __attribute__((weak)) on-line, I thought, there may be an easier way to achieve this. But, I quite possibly misunderstood the use of that attribute. It sounds like you're not using dlsym as intended, then. If you used: void (*my_function)(void); my_function = dlsym(handle, target_function); if (my_function == NULL) oops(); (*my_function)(); Hi, That is what I am doing right now. This method, allows the main program to compile just fine as the storage is reserved for the my_function (in your example). While, I had initially hoped one could have objects (e.g., int types, or structs) defined in the yet-to-be-loaded .so object file, referenced in the main program sources, and be able get the main program to link; then at run time, the .so object is loaded, and symbols updated/ resolved, just like the (system) dynamic linker (implicitly) does at program start up. Thanks for all your replies! --patrick
panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
Hi,
Not sure if anyone is interested in this panic as the
snapshot is a bit old.
Happend while gdb-ing a process.
(typing off images; expect typos)
login: panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0
failed: file ../../../../lock.c, line 126
Stopped at Debugger+0x5: leave
RUN AT LEAST ...
ddb{0} trace
Debugger() at Debugger+0x5
panic() at panic+0xee
__assert() at __assert+0x21
_kernel_lock_init() at _kernel_lock_init
issignal() at issignal+0x205
sleep_setup_signal() at sleep_setup_signal+0x94
tsleep() at tsleep+0x92
thrsleep() at thrsleep+0x1b5
sys___thrsleep() at sys___thrsleep+0x6f
syscall() at syscall+0x249
--- syscall (number 94) ---
end of kernel
end trace frame: 0x1d47e3209d80, count: -10
0x1d47e4eba95a:
ddb{0}
now, the ps has a lot of output. a lot of firefox and gimp, and...
I'm happy to email photos to anyone interested. But I think these
are the interesting parts:
ddb{0} ps
PID PPIDPGRPUID S FLAGS WAITCOMMAND
2944 154772944 100030x83 waitgdb
14290 2944 30442 10002 0x4100603 myapp
5416 2944 30442 10002 0x4100603 myapp
7295 2944 30442 10002 0x4100603 myapp
17989 2944 30442 10002 0x4100603 myapp
21860 2944 30442 10002 0x4100603 myapp
25934 2944 30442 10002 0x4100603 myapp
32131 2944 30442 10002 0x4100603 myapp
4616 2944 30442 10002 0x4100603 myapp
17063 2944 30442 10002 0xc100603 myapp
*26101 2944 30442 10007 0xc100683 thrsleepmyapp
30442 2944 30442 10003 0x8000683 pollmyapp
3142 93373142 10004 0x8000403 vi
11005 8117 27631 10003 0x4100082 pollfirefox
... lots of firefox threads ...
then other stuff ...
After typing all that: *ga!* This shit is in the dmesg :P
myapp is what was being debugged.
The app dlopen()'s a .so object, looks for a object holding
pointers to functions, and executes functions upon events.
I had set a break point in one of those functions, or rather
I thought I did ... then boom!
Interesting?
Side note 1:
Incidentally, I couldn't figure out how to correctly leave
a symbol unresolved while compiling myapp, until dlopen()
time. If anyone cares to give me a pointer (ha!) off-list,
I'd much appreciate it. I don't know if this is the cause
of the problem, but I'd have thunk, error like this, would
just crash the app not the system. The loaded functions
seem to execute just fine though I see log statements
from them.
Side note 2:
I see gdb crash quite often on subsequent run of app being
debugged. I've noticed this on a bunch of snapshots dating
some months back. Never reported it cause, I don't have a
simple test-case to submit, and things have been moving
fast with what you guys are doing.
$ gdb someapp
(gdb) break paris
(gdb) r args
[then break point]
[examine ...]
(gdb) p inside_wine_glass
[... then let it run to end]
(gdb) c
[start again without quitting gdb!]
(gdb) r args
[then break point]
[examine ...]
(gdb) p wine_bottle
gdb coredumps!
Cheers,
--patrick
sd2: 15322MB, 512 bytes/sector, 31379456 sectors
sd2 detached
scsibus2 detached
umass1 detached
panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0 failed:
file ../../../../kern/kern_lock.c, line 126
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu #' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb{0} Debugger() at Debugger+0x5
panic() at panic+0xee
__assert() at __assert+0x21
_kernel_lock_init() at _kernel_lock_init
issignal() at issignal+0x205
sleep_setup_signal() at sleep_setup_signal+0x94
tsleep() at tsleep+0x92
thrsleep() at thrsleep+0x1b5
sys___thrsleep() at sys___thrsleep+0x6f
syscall() at syscall+0x249
--- syscall (number 94) ---
end of kernel
end trace frame: 0x1d47e3209d80, count: -10
0x1d47e4eba95a:
ddb{0}PID PPID PGRPUID S FLAGS WAIT COMMAND
2944 15477 2944 1000 30x83 wait gdb
14290 2944 30442 1000 2 0x4100603myapp
5416 2944 30442 1000 2 0x4100603myapp
7295 2944 30442 1000 2 0x4100603myapp
17989 2944 30442 1000 2 0x4100603myapp
21860 2944 30442 1000 2 0x4100603myapp
25934 2944 30442 1000 2 0x4100603myapp
32131 2944
Re: panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
On Tue, Jan 28, 2014 at 4:55 PM, patrick keshishian sids...@boxsoft.com wrote: Not sure if anyone is interested in this panic as the snapshot is a bit old. Happend while gdb-ing a process. (typing off images; expect typos) login: panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0 failed: file ../../../../lock.c, line 126 That's the bug I was working on at the start of the n2k14 hackathon. I've managed to simplify some of the bits underneath it, so maybe I can get the intermediate problem (recursive tsleep() usage) resolved... Side note 1: Incidentally, I couldn't figure out how to correctly leave a symbol unresolved while compiling myapp, until dlopen() time. If you can include the shared-object/library in the link command line, then the symbol can be referenced at link time. That lets you refer to the symbols directly as if they were in the executable itself, though you must still declare them for the compiler to know what type they are, etc. If you need to delay the loading of the shared-object until after process startup via dlopen(), then the executable should use dlsym() to get the symbol address, passing it the handle that dlopen() returned. c.f. the dlsym(3) manpage for details and other possbilities. Side note 2: I see gdb crash quite often on subsequent run of app being debugged. I've noticed this on a bunch of snapshots dating some months back. Never reported it cause, I don't have a simple test-case to submit, and things have been moving fast with what you guys are doing. gdb has had many bugs in the past. Many are fixed in versions newer than the version that are shipped with OpenBSD, but licensing keeps us from including them in general. :-/ Anyone care to hack on gdb (no copying GPLv3 source!)? Philip Guenther
Re: panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
On Tue, Jan 28, 2014 at 09:02:31PM -0800, Philip Guenther wrote: On Tue, Jan 28, 2014 at 4:55 PM, patrick keshishian sids...@boxsoft.com wrote: Not sure if anyone is interested in this panic as the snapshot is a bit old. Happend while gdb-ing a process. (typing off images; expect typos) login: panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0 failed: file ../../../../lock.c, line 126 That's the bug I was working on at the start of the n2k14 hackathon. I've managed to simplify some of the bits underneath it, so maybe I can get the intermediate problem (recursive tsleep() usage) resolved... Thanks. Side note 1: Incidentally, I couldn't figure out how to correctly leave a symbol unresolved while compiling myapp, until dlopen() time. If you can include the shared-object/library in the link command line, then the symbol can be referenced at link time. That lets you refer to the symbols directly as if they were in the executable itself, though you must still declare them for the compiler to know what type they are, etc. A bit of a chicken-and-egg situation for this case. Unless I do some ugly hopping around in Makefiles, it would be difficult to include the object on the link line for the main program. If you need to delay the loading of the shared-object until after process startup via dlopen(), then the executable should use dlsym() to get the symbol address, passing it the handle that dlopen() returned. Yes. I do that right now with a bit of dance. However, reading a bit about __attribute__((weak)) on-line, I thought, there may be an easier way to achieve this. But, I quite possibly misunderstood the use of that attribute. Thanks again, --patrick c.f. the dlsym(3) manpage for details and other possbilities. Side note 2: I see gdb crash quite often on subsequent run of app being debugged. I've noticed this on a bunch of snapshots dating some months back. Never reported it cause, I don't have a simple test-case to submit, and things have been moving fast with what you guys are doing. gdb has had many bugs in the past. Many are fixed in versions newer than the version that are shipped with OpenBSD, but licensing keeps us from including them in general. :-/ Anyone care to hack on gdb (no copying GPLv3 source!)? Philip Guenther
Re: panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
On Tue, Jan 28, 2014 at 10:31 PM, patrick keshishian sids...@boxsoft.com wrote: On Tue, Jan 28, 2014 at 09:02:31PM -0800, Philip Guenther wrote: .. If you need to delay the loading of the shared-object until after process startup via dlopen(), then the executable should use dlsym() to get the symbol address, passing it the handle that dlopen() returned. Yes. I do that right now with a bit of dance. However, reading a bit about __attribute__((weak)) on-line, I thought, there may be an easier way to achieve this. But, I quite possibly misunderstood the use of that attribute. Undefined weak symbols are useful for testing for the presence of optional functionality loaded at the same time as the code with the reference to the weak symbol. If the code that may contain the symbol will be loaded later, using a weak symbol reference is fragile, as it is dependent on the order of operations and reputably can conflict with compiler optimizations on some platforms. Even Sun, who were quite in love with their dynamic linker, recommended against doing that (c.f. the Sun Linker and Libraries Guide). Philip Guenther
Re: panic: __mp_lock_held(sched_lock) 2013-DEC-27 snapshot
patrick keshishian sids...@boxsoft.com writes: On Tue, Jan 28, 2014 at 09:02:31PM -0800, Philip Guenther wrote: On Tue, Jan 28, 2014 at 4:55 PM, patrick keshishian sids...@boxsoft.com wrote: Not sure if anyone is interested in this panic as the snapshot is a bit old. Happend while gdb-ing a process. (typing off images; expect typos) login: panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0 failed: file ../../../../lock.c, line 126 That's the bug I was working on at the start of the n2k14 hackathon. I've managed to simplify some of the bits underneath it, so maybe I can get the intermediate problem (recursive tsleep() usage) resolved... Thanks. Side note 1: Incidentally, I couldn't figure out how to correctly leave a symbol unresolved while compiling myapp, until dlopen() time. If you can include the shared-object/library in the link command line, then the symbol can be referenced at link time. That lets you refer to the symbols directly as if they were in the executable itself, though you must still declare them for the compiler to know what type they are, etc. A bit of a chicken-and-egg situation for this case. Unless I do some ugly hopping around in Makefiles, it would be difficult to include the object on the link line for the main program. If you need to delay the loading of the shared-object until after process startup via dlopen(), then the executable should use dlsym() to get the symbol address, passing it the handle that dlopen() returned. Yes. I do that right now with a bit of dance. However, reading a bit about __attribute__((weak)) on-line, I thought, there may be an easier way to achieve this. But, I quite possibly misunderstood the use of that attribute. It sounds like you're not using dlsym as intended, then. If you used: void (*my_function)(void); my_function = dlsym(handle, target_function); if (my_function == NULL) oops(); (*my_function)(); then your program wouldn't need to reference target_function at all. [...] -- jca | PGP: 0x1524E7EE / 5135 92C1 AD36 5293 2BDF DDCC 0DFA 74AE 1524 E7EE (previous: 0x06A11494 / 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494)
panic __mp_lock_held(sched_lock)
Hi,
On my ThinkPat T430s I am trying to debug multithreaded qemu by
attaching gdb. This crashes the kernel of the host system within
a few minutes. Luckily I managed to attach a serial over lan with
Intel AMT.
login: panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0
failed: file ../../../../kern/kern_lock.c, line 126
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu #' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb{0} trace
Debugger() at Debugger+0x5
panic() at panic+0xee
__assert() at __assert+0x21
_kernel_lock_init() at _kernel_lock_init
issignal() at issignal+0x205
sleep_setup_signal() at sleep_setup_signal+0x94
tsleep() at tsleep+0x86
sys_sigsuspend() at sys_sigsuspend+0x46
syscall() at syscall+0x249
--- syscall (number 111) ---
end of kernel
end trace frame: 0x7fe55fdbef0, count: -9
0x7fe50c4cdcc:
PID PPID PGRPUID S FLAGS WAIT COMMAND
28380 5201 5983 1000 3 0x4100080 thrsleep qemu-system-x86_
* 5825 5201 5983 1000 7 0xc100088 pause qemu-system-x86_
18891 5201 5983 1000 3 0xc100080 sigwait qemu-system-x86_
5983 5201 5983 1000 3 0x880 thrsleep qemu-system-x86_
19446 22621 19446 1000 30x80 poll gdb
5201 12983 5201 1000 30x80 wait gdb
The kernel lock is acquired in mi_syscall() as sys_sigsuspend() needs
it. tsleep() calls sleep_setup() which acquires the sched lock.
Then sleep_setup_signal() calls issignal() via the macro CURSIG().
The function issignal() is full of side effects, especially for a
traced process.
There the kernel lock is acquired again, which should be fine as
it is a recursive lock. But to avoid deadlocks, _kernel_lock()
asserts that is is acquired before sched lock. This check is too
strict, the condition is only true when the lock is taken the first
time.
Index: kern/kern_lock.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/kern/kern_lock.c,v
retrieving revision 1.42
diff -u -p -u -p -r1.42 kern_lock.c
--- kern/kern_lock.c6 May 2013 16:37:55 - 1.42
+++ kern/kern_lock.c11 Aug 2013 01:54:06 -
@@ -123,7 +123,10 @@ _kernel_lock_init(void)
void
_kernel_lock(void)
{
- SCHED_ASSERT_UNLOCKED();
+#ifdef DIAGNOSTIC
+ if (__mp_lock_held(kernel_lock) == 0)
+ SCHED_ASSERT_UNLOCKED();
+#endif /* DIAGNOSTIC */
__mp_lock(kernel_lock);
}
Unfortunately this fix does not solve my problem. With that I get
another panic: wakeup: p_stat is 7
login: panic: wakeup: p_stat is 7
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu #' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb{3} trace
Debugger() at Debugger+0x5
panic() at panic+0xee
wakeup_n() at wakeup_n+0xfd
sys___thrwakeup() at sys___thrwakeup+0x54
syscall() at syscall+0x249
--- syscall (number 301) ---
end of kernel
end trace frame: 0x684cb9237c0, count: -5
0x684bf834c2a:
PID PPID PGRPUID S FLAGS WAIT COMMAND
10959 11922 10959 1000 30x80 wait gdb
*11287 10959 10043 1000 7 0xc10qemu-system-x86_
11131 10959 10043 1000 3 0xc100080 sigwait qemu-system-x86_
10043 10959 10043 1000 7 0x800qemu-system-x86_
I will investigate further.
bluhm
Re: panic __mp_lock_held(sched_lock)
Date: Sun, 11 Aug 2013 11:46:29 +0200
From: Alexander Bluhm alexander.bl...@gmx.net
Hi,
On my ThinkPat T430s I am trying to debug multithreaded qemu by
attaching gdb. This crashes the kernel of the host system within
a few minutes. Luckily I managed to attach a serial over lan with
Intel AMT.
login: panic: kernel diagnostic assertion __mp_lock_held(sched_lock) == 0
failed: file ../../../../kern/kern_lock.c, line 126
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu #' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb{0} trace
Debugger() at Debugger+0x5
panic() at panic+0xee
__assert() at __assert+0x21
_kernel_lock_init() at _kernel_lock_init
issignal() at issignal+0x205
sleep_setup_signal() at sleep_setup_signal+0x94
tsleep() at tsleep+0x86
sys_sigsuspend() at sys_sigsuspend+0x46
syscall() at syscall+0x249
--- syscall (number 111) ---
end of kernel
end trace frame: 0x7fe55fdbef0, count: -9
0x7fe50c4cdcc:
PID PPID PGRPUID S FLAGS WAIT COMMAND
28380 5201 5983 1000 3 0x4100080 thrsleep qemu-system-x86_
* 5825 5201 5983 1000 7 0xc100088 pause qemu-system-x86_
18891 5201 5983 1000 3 0xc100080 sigwait qemu-system-x86_
5983 5201 5983 1000 3 0x880 thrsleep qemu-system-x86_
19446 22621 19446 1000 30x80 poll gdb
5201 12983 5201 1000 30x80 wait gdb
The kernel lock is acquired in mi_syscall() as sys_sigsuspend() needs
it. tsleep() calls sleep_setup() which acquires the sched lock.
Then sleep_setup_signal() calls issignal() via the macro CURSIG().
The function issignal() is full of side effects, especially for a
traced process.
There the kernel lock is acquired again, which should be fine as
it is a recursive lock. But to avoid deadlocks, _kernel_lock()
asserts that is is acquired before sched lock. This check is too
strict, the condition is only true when the lock is taken the first
time.
I don't think that check is too strict. Rather the ptrace(2)
functionality is subtly broken.
Index: kern/kern_lock.c
===
RCS file: /data/mirror/openbsd/cvs/src/sys/kern/kern_lock.c,v
retrieving revision 1.42
diff -u -p -u -p -r1.42 kern_lock.c
--- kern/kern_lock.c 6 May 2013 16:37:55 - 1.42
+++ kern/kern_lock.c 11 Aug 2013 01:54:06 -
@@ -123,7 +123,10 @@ _kernel_lock_init(void)
void
_kernel_lock(void)
{
- SCHED_ASSERT_UNLOCKED();
+#ifdef DIAGNOSTIC
+ if (__mp_lock_held(kernel_lock) == 0)
+ SCHED_ASSERT_UNLOCKED();
+#endif /* DIAGNOSTIC */
__mp_lock(kernel_lock);
}
Unfortunately this fix does not solve my problem. With that I get
another panic: wakeup: p_stat is 7
login: panic: wakeup: p_stat is 7
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu #' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb{3} trace
Debugger() at Debugger+0x5
panic() at panic+0xee
wakeup_n() at wakeup_n+0xfd
sys___thrwakeup() at sys___thrwakeup+0x54
syscall() at syscall+0x249
--- syscall (number 301) ---
end of kernel
end trace frame: 0x684cb9237c0, count: -5
0x684bf834c2a:
PID PPID PGRPUID S FLAGS WAIT COMMAND
10959 11922 10959 1000 30x80 wait gdb
*11287 10959 10043 1000 7 0xc10qemu-system-x86_
11131 10959 10043 1000 3 0xc100080 sigwait qemu-system-x86_
10043 10959 10043 1000 7 0x800qemu-system-x86_
I will investigate further.
bluhm
