Re: Swapfile problem in 6?
On 2 Jan, Lars Kristiansen wrote: Attempting to catch up with my backlog of unread email, only 12K unread messages to go ... On 24 Nov, Rob wrote: I have cvsup'ed the sources to STABLE as of Nov. 23rd 2005. After recompiling/installing world and debug-kernel, I again get a kernel deadlock when using swapfile: http://surfion.snu.ac.kr/~lahaye/swapfile2.txt Previous deadlocks are still documented here http://surfion.snu.ac.kr/~lahaye/swapfile.txt I hope this is of use for fixing this bug in 6. If further investigation is needed, then please let me know. This is a deadlock caused by memory exhaustion. The pagedaemon only has a limited number of bufs that it uses for writing dirty pages to swap to prevent it from saturating the I/O subsystem with large numbers of writes. In this case, pagedaemon is trying to free up memory by writing dirty pages, and it has used up all of its bufs and is waiting for the write requests to complete and the bufs the bufs to be returned to it. This isn't happening because md0 is stuck waiting for memory. This is a little bit suprising to me because it looks like writes to vnode backed devices are done synchronously by default. If you have a chance to test this again, a stack trace of md0 in the deadlock state would be interesting. I'd like to know where md0 is getting stuck. I wonder if pagedaemon should scan ahead and more agressively discard clean pages when it has run out of bufs to write dirty pages, especially in low memory situations. Preventing the creation of more dirty pages would be nice, but I don't know how to do that ... Just in case it can help. Do not have this machine available for testing at the moment but this is the last debuginfo I did get from it. Here is a trace from a situation when a possible idle system got stuck during the night and db showed only one locked vnode: db show lockedvnods Locked vnodes 0xc1309330: tag ufs, type VREG usecount 1, writecount 1, refcount 154 mountedhere 0 flags () v_object 0xc12cb39c ref 0 pages 606 lock type ufs: EXCL (count 1) by thread 0xc126b900 (pid 178) ino 8155, on dev ad0s1f db trace 178 Tracing pid 178 tid 100058 td 0xc126b900 sched_switch(c126b900,0,1) at 0xc066a4db = sched_switch+0x17b mi_switch(1,0) at 0xc065f49e = mi_switch+0x27e sleepq_switch(c09b2a98,c484bacc,c065f0e3,c09b2a98,0) at 0xc0677f00 = sleepq_switch+0xe0 sleepq_wait(c09b2a98,0,0,c08ad92d,37b) at 0xc0678100 = sleepq_wait+0x30 msleep(c09b2a98,c09b2d00,244,c08adb6a,0) at 0xc065f0e3 = msleep+0x333 vm_wait(c12cb39c,0,c08990f3,ad7,c06512a4) at 0xc07c6a71 = vm_wait+0x91 allocbuf(c28fa9d8,4000,354000,0,354000) at 0xc06a2f89 = allocbuf+0x4e9 getblk(c1309330,d5,0,4000,0) at 0xc06a29cb = getblk+0x4eb cluster_read(c1309330,1000,0,d5,0) at 0xc06a5d65 = cluster_read+0xe5 ffs_read(c484bc9c) at 0xc07a631f = ffs_read+0x28f VOP_READ_APV(c09309a0,c484bc9c) at 0xc0838aab = VOP_READ_APV+0x7b mdstart_vnode(c1310800,c1634294,c1310820,1,c0566e10) at 0xc056688c = mdstart_vnode+0xec md_kthread(c1310800,c484bd38,c1310800,c0566e10,0) at 0xc0566f7f = md_kthread+0x16f fork_exit(c0566e10,c1310800,c484bd38) at 0xc0645618 = fork_exit+0xa8 fork_trampoline() at 0xc0816f3c = fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xc484bd6c, ebp = 0 --- The md thread is stuck waiting for memory to be freed by pagedaemon. Pagedaemon is stuck waiting for at least one of its pageout requests to complete. The pageout requests are probably all stuck waiting for md. I had expected that the problem is that while pagedaemon is allowed to dig deeper into the free page pool, I didn't think that the md thread would be allowed to, allowing the md thread to get wedged first. That does not appear to be the case because the vm_page_alloc() call in allocbuf() has the VM_ALLOC_SYSTEM flag set, which should match vm_page_alloc()'s treatment of requests by pagedaemon. I don't see how the md thread could be consuming a large number of reserved pages, but it looks like that must be what is happening. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Attempting to catch up with my backlog of unread email, only 12K unread messages to go ... On 24 Nov, Rob wrote: I have cvsup'ed the sources to STABLE as of Nov. 23rd 2005. After recompiling/installing world and debug-kernel, I again get a kernel deadlock when using swapfile: http://surfion.snu.ac.kr/~lahaye/swapfile2.txt Previous deadlocks are still documented here http://surfion.snu.ac.kr/~lahaye/swapfile.txt I hope this is of use for fixing this bug in 6. If further investigation is needed, then please let me know. This is a deadlock caused by memory exhaustion. The pagedaemon only has a limited number of bufs that it uses for writing dirty pages to swap to prevent it from saturating the I/O subsystem with large numbers of writes. In this case, pagedaemon is trying to free up memory by writing dirty pages, and it has used up all of its bufs and is waiting for the write requests to complete and the bufs the bufs to be returned to it. This isn't happening because md0 is stuck waiting for memory. This is a little bit suprising to me because it looks like writes to vnode backed devices are done synchronously by default. If you have a chance to test this again, a stack trace of md0 in the deadlock state would be interesting. I'd like to know where md0 is getting stuck. I wonder if pagedaemon should scan ahead and more agressively discard clean pages when it has run out of bufs to write dirty pages, especially in low memory situations. Preventing the creation of more dirty pages would be nice, but I don't know how to do that ... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On 17 Nov, Kris Kennaway wrote: On Thu, Nov 17, 2005 at 04:33:50PM -0800, Rob wrote: --- Kris Kennaway [EMAIL PROTECTED] wrote: I commented on it elsewhere in this thread. Do you mean your comment on the swap_pager error: Quote: AFAICT that is just a trigger-happy timer..it's supposed to detect when a swap operation took too long to complete, but it also triggers on swapfiles since they're so much less efficient (i.e. slower) than swapping onto a bare device. EndQuote. Right: harmless warning not error. This isn't totally harmless because the pagedaemon is only allowed a handful of outstanding writes. It gets stuck when it runs out. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Attempting to catch up with my backlog of unread email, only 12K unread messages to go ... On 24 Nov, Rob wrote: I have cvsup'ed the sources to STABLE as of Nov. 23rd 2005. After recompiling/installing world and debug-kernel, I again get a kernel deadlock when using swapfile: http://surfion.snu.ac.kr/~lahaye/swapfile2.txt Previous deadlocks are still documented here http://surfion.snu.ac.kr/~lahaye/swapfile.txt I hope this is of use for fixing this bug in 6. If further investigation is needed, then please let me know. This is a deadlock caused by memory exhaustion. The pagedaemon only has a limited number of bufs that it uses for writing dirty pages to swap to prevent it from saturating the I/O subsystem with large numbers of writes. In this case, pagedaemon is trying to free up memory by writing dirty pages, and it has used up all of its bufs and is waiting for the write requests to complete and the bufs the bufs to be returned to it. This isn't happening because md0 is stuck waiting for memory. This is a little bit suprising to me because it looks like writes to vnode backed devices are done synchronously by default. If you have a chance to test this again, a stack trace of md0 in the deadlock state would be interesting. I'd like to know where md0 is getting stuck. I wonder if pagedaemon should scan ahead and more agressively discard clean pages when it has run out of bufs to write dirty pages, especially in low memory situations. Preventing the creation of more dirty pages would be nice, but I don't know how to do that ... Just in case it can help. Do not have this machine available for testing at the moment but this is the last debuginfo I did get from it. Here is a trace from a situation when a possible idle system got stuck during the night and db showed only one locked vnode: db show lockedvnods Locked vnodes 0xc1309330: tag ufs, type VREG usecount 1, writecount 1, refcount 154 mountedhere 0 flags () v_object 0xc12cb39c ref 0 pages 606 lock type ufs: EXCL (count 1) by thread 0xc126b900 (pid 178) ino 8155, on dev ad0s1f db trace 178 Tracing pid 178 tid 100058 td 0xc126b900 sched_switch(c126b900,0,1) at 0xc066a4db = sched_switch+0x17b mi_switch(1,0) at 0xc065f49e = mi_switch+0x27e sleepq_switch(c09b2a98,c484bacc,c065f0e3,c09b2a98,0) at 0xc0677f00 = sleepq_switch+0xe0 sleepq_wait(c09b2a98,0,0,c08ad92d,37b) at 0xc0678100 = sleepq_wait+0x30 msleep(c09b2a98,c09b2d00,244,c08adb6a,0) at 0xc065f0e3 = msleep+0x333 vm_wait(c12cb39c,0,c08990f3,ad7,c06512a4) at 0xc07c6a71 = vm_wait+0x91 allocbuf(c28fa9d8,4000,354000,0,354000) at 0xc06a2f89 = allocbuf+0x4e9 getblk(c1309330,d5,0,4000,0) at 0xc06a29cb = getblk+0x4eb cluster_read(c1309330,1000,0,d5,0) at 0xc06a5d65 = cluster_read+0xe5 ffs_read(c484bc9c) at 0xc07a631f = ffs_read+0x28f VOP_READ_APV(c09309a0,c484bc9c) at 0xc0838aab = VOP_READ_APV+0x7b mdstart_vnode(c1310800,c1634294,c1310820,1,c0566e10) at 0xc056688c = mdstart_vnode+0xec md_kthread(c1310800,c484bd38,c1310800,c0566e10,0) at 0xc0566f7f = md_kthread+0x16f fork_exit(c0566e10,c1310800,c484bd38) at 0xc0645618 = fork_exit+0xa8 fork_trampoline() at 0xc0816f3c = fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xc484bd6c, ebp = 0 --- -- Lars ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Peter Jeremy wrote: On Thu, 2005-Nov-17 00:00:03 -0800, Rob wrote: The only way I know of how to trigger the deadlock, is to compile a new kernel and the 'linking kernel' stage will lock-up the PC. With a regular kernel, this takes 2.5 hours until deadlock, but with a fully equipped debug kernel it takes about 8 hours When the first deadlock occurs, you have a fully populated set of kernel objects (though possibly some of them are in the buffer case rather than on disk). You should be able to quickly reproduce the panic by running: # cd /usr/obj/usr/src/sys/KERNEL # make I have cvsup'ed the sources to STABLE as of Nov. 23rd 2005. After recompiling/installing world and debug-kernel, I again get a kernel deadlock when using swapfile: http://surfion.snu.ac.kr/~lahaye/swapfile2.txt Previous deadlocks are still documented here http://surfion.snu.ac.kr/~lahaye/swapfile.txt I hope this is of use for fixing this bug in 6. If further investigation is needed, then please let me know. Regards, Rob. __ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Rob wrote: --- Peter Jeremy PeterJeremy at optushome.com.au wrote: Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot If in doubt, post the output from the above commands here and someone will hopefully provide further input. The output is here: http://surfion.snu.ac.kr/~lahaye/swapfile.txt Has this kernel debug output been of any use? Or was this analysis not helping much as to why 6 has problems with swapfiles? I can dig further into the problem, if I get some more hints how to do so. Regards, Rob. __ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Lars Kristiansen wrote: Hello again, I am the me too-guy with console-access. Hi Lars, I'm the one who started this thread, when I could not compile a new kernel after I upgraded to 6 (Pentium-1, 150 MHz, 32 MB Ram). How do you trigger the deadlock? The only way I know of how to trigger the deadlock, is to compile a new kernel and the 'linking kernel' stage will lock-up the PC. With a regular kernel, this takes 2.5 hours until deadlock, but with a fully equipped debug kernel it takes about 8 hours If you have a different (and possibly quicker) way to trigger the deadlock, I would like to try that also on my system. Please let me know. Regards, Rob. - Yahoo! FareChase - Search multiple travel sites in one click. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Thu, 2005-Nov-17 00:00:03 -0800, Rob wrote:
The only way I know of how to trigger the deadlock, is to compile
a new kernel and the 'linking kernel' stage will lock-up the PC.
With a regular kernel, this takes 2.5 hours until deadlock, but with
a fully equipped debug kernel it takes about 8 hours
When the first deadlock occurs, you have a fully populated set of kernel
objects (though possibly some of them are in the buffer case rather than
on disk). You should be able to quickly reproduce the panic by running:
# cd /usr/obj/usr/src/sys/KERNEL
# make
(Adjust the directory to suit your config name and MAKEOBJDIRPREFIX).
Alternatively, check out the following lines in /usr/src/Makefile.inc1
# -DNO_KERNELCONFIG do not run config in ${MAKE} buildkernel
# -DNO_KERNELCLEAN do not run ${MAKE} clean in ${MAKE} buildkernel
# -DNO_KERNELDEPEND do not run ${MAKE} depend in ${MAKE} buildkernel
--
Peter Jeremy
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Peter Jeremy wrote: On Thu, 2005-Nov-17 00:00:03 -0800, Rob wrote: The only way I know of how to trigger the deadlock, is to compile a new kernel and the 'linking kernel' stage will lock-up the PC. With a regular kernel, this takes 2.5 hours until deadlock, but with a fully equipped debug kernel it takes about 8 hours When the first deadlock occurs, you have a fully populated set of kernel objects (though possibly some of them are in the buffer case rather than on disk). You should be able to quickly reproduce the panic by running: # cd /usr/obj/usr/src/sys/KERNEL # make I'm doing this right now... Although I have not got into the deadlock yet, I do get these lines in the serial console: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 12333, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22990, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22928, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22625, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22990, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 22784, size: 8192 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 23233, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 23404, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 23154, size: 4096 A typical snapshot of 'swapinfo' at this time is: Device 1K-blocks UsedAvail Capacity /dev/ad0s1b 398483984039848 100% /dev/md0 13107254136 13107241% Total 170920939767694455% (By the way, notice that the 'Avail' column of /dev/ad0s1b and /dev/md0 is wrong; for Total it's correct. Is this a general bug in 6 or only on my PC? ) Googling on the above swap_pager lines, I found this conversation: http://lists.freebsd.org/pipermail/freebsd-current/2005-September/00.html which claims that DragonFly has fixed this problem. Regards, Rob. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Lars Kristiansen wrote: Hello again, I am the me too-guy with console-access. Hi Lars, I'm the one who started this thread, when I could not compile a new kernel after I upgraded to 6 (Pentium-1, 150 MHz, 32 MB Ram). How do you trigger the deadlock? The only way I know of how to trigger the deadlock, is to compile a new kernel and the 'linking kernel' stage will lock-up the PC. With a regular kernel, this takes 2.5 hours until deadlock, but with a fully equipped debug kernel it takes about 8 hours If you have a different (and possibly quicker) way to trigger the deadlock, I would like to try that also on my system. Please let me know. Hi Rob! swapoff all other swap than md0, which is on the same disk as busy filesystems. Then I run for example: nice +19 portupgrade -frbWp libtool Did you see the debug-output I posted? Did those tell you anything? I do not understand those things. But I do have the debugscreen open after a crash if you want me too run anything. Just glad if I can be of help. -- Lars Regards, Rob. - Yahoo! FareChase - Search multiple travel sites in one click. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
--- Peter Jeremy [EMAIL PROTECTED] wrote: Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot If in doubt, post the output from the above commands here and someone will hopefully provide further input. The output is here: http://surfion.snu.ac.kr/~lahaye/swapfile.txt A swapfile is only swap device: /swapfile of 128 MB on /dev/ad0s1a as /dev/md0 At time of deadlock: swapdevice used 13% Serial console gets exactly this line every few seconds: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4317, size: 4096 Do you understand this? Rob. __ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Thu, Nov 17, 2005 at 02:52:12PM -0800, Rob wrote: --- Peter Jeremy [EMAIL PROTECTED] wrote: Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot If in doubt, post the output from the above commands here and someone will hopefully provide further input. The output is here: http://surfion.snu.ac.kr/~lahaye/swapfile.txt A swapfile is only swap device: /swapfile of 128 MB on /dev/ad0s1a as /dev/md0 At time of deadlock: swapdevice used 13% Serial console gets exactly this line every few seconds: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 4317, size: 4096 Do you understand this? I commented on it elsewhere in this thread. Kris pgpeguxnEhKqJ.pgp Description: PGP signature
Re: Swapfile problem in 6?
--- Kris Kennaway [EMAIL PROTECTED] wrote: I commented on it elsewhere in this thread. Do you mean your comment on the swap_pager error: Quote: AFAICT that is just a trigger-happy timer..it's supposed to detect when a swap operation took too long to complete, but it also triggers on swapfiles since they're so much less efficient (i.e. slower) than swapping onto a bare device. EndQuote. Remains the question: using this swapfile always worked very well with 5-Stable, but suddenly with 6 I run into these troubles. I hoped that the debugger output made some sense to an expert as to why this happens with 6, but never with 5 Rob. __ Start your day with Yahoo! - Make it your home page! http://www.yahoo.com/r/hs ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Thu, Nov 17, 2005 at 04:33:50PM -0800, Rob wrote: --- Kris Kennaway [EMAIL PROTECTED] wrote: I commented on it elsewhere in this thread. Do you mean your comment on the swap_pager error: Quote: AFAICT that is just a trigger-happy timer..it's supposed to detect when a swap operation took too long to complete, but it also triggers on swapfiles since they're so much less efficient (i.e. slower) than swapping onto a bare device. EndQuote. Right: harmless warning not error. Remains the question: using this swapfile always worked very well with 5-Stable, but suddenly with 6 I run into these troubles. I hoped that the debugger output made some sense to an expert as to why this happens with 6, but never with 5 Give it a little time for someone to take a look. Kris pgpRhAJFn7wpW.pgp Description: PGP signature
Re: Swapfile problem in 6?
--- Peter Jeremy [EMAIL PROTECTED] wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. First question is: Does the system still deadlock? INVARIANTS and WITNESS will have added sanity checks which might have picked up the problem. Indeed, I have this in my DEBUG kernel config: makeoptions DEBUG=-g options INVARIANT_SUPPORT options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB options BREAK_TO_DEBUGGER and the deadlock disappears. Does that already give a hint what's wrong? If not, then what should I remove/keep from the above list, to allow the deadlock to reappear and still be able to debug the problem? According to your info above, I would say I'd remove INVARIANT_SUPPORT, INVARIANTS, WITNESS and WITNESS_KDB; leave the others in there and try again with such a debug kernel. Right? Thanks, Rob. Rob. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. First question is: Does the system still deadlock? INVARIANTS and WITNESS will have added sanity checks which might have picked up the problem. 1) Can I debug a kernel that does not crash, but just hangs in a deadlock? Everything seems to be frozen, except pinging the PC Have a look at http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-online-ddb.html and ddb(4). Unless you have another system handy, you might like to print out ddb(4) - it's difficult to read man pages when you're in the kernel debugger :-). 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters - which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot - see http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-gdb.html If in doubt, post the output from the above commands here and someone will hopefully provide further input. Hello again, I am the me too-guy with console-access. I am not a programmer and it is the first time I see debugging screen. It deadlocked again, and I did as advised above: (ddb: show lockedvnods; ps ; panic) but did not understand much of the output. Looked maybe like syncer and swap_pager was locked? Do i need to write all this down or can I get the output saved somewhere? I got a 32MB coredump but the same lack of understanding applies. Please tell me if I can be of any help! This is fun. -- Lars -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Wed, Nov 16, 2005 at 04:01:36PM +0100, Lars Kristiansen wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. First question is: Does the system still deadlock? INVARIANTS and WITNESS will have added sanity checks which might have picked up the problem. 1) Can I debug a kernel that does not crash, but just hangs in a deadlock? Everything seems to be frozen, except pinging the PC Have a look at http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-online-ddb.html and ddb(4). Unless you have another system handy, you might like to print out ddb(4) - it's difficult to read man pages when you're in the kernel debugger :-). 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters - which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot - see http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-gdb.html If in doubt, post the output from the above commands here and someone will hopefully provide further input. Hello again, I am the me too-guy with console-access. I am not a programmer and it is the first time I see debugging screen. It deadlocked again, and I did as advised above: (ddb: show lockedvnods; ps ; panic) but did not understand much of the output. Looked maybe like syncer and swap_pager was locked? Do i need to write all this down or can I get the output saved somewhere? I got a 32MB coredump but the same lack of understanding applies. Please tell me if I can be of any help! This is fun. Do you have the ability to connect another computer by RS-232? It's easy to get a serial terminal console going (err that is if you find the right guide as opposed to stabbing blindly and just referencing man pages as I like to do.) The coredump should supply the same (and more) information, and someone can walk through with you doing a post-mortem gdb session. For example, try doing the following now that you have the coredump: # ps wwwauxlH -N /boot/kernel/kernel -M /var/crash/vmcore.whatever -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Wed, Nov 16, 2005 at 04:01:36PM +0100, Lars Kristiansen wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. First question is: Does the system still deadlock? INVARIANTS and WITNESS will have added sanity checks which might have picked up the problem. 1) Can I debug a kernel that does not crash, but just hangs in a deadlock? Everything seems to be frozen, except pinging the PC Have a look at http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-online-ddb.html and ddb(4). Unless you have another system handy, you might like to print out ddb(4) - it's difficult to read man pages when you're in the kernel debugger :-). 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters - which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot - see http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-gdb.html If in doubt, post the output from the above commands here and someone will hopefully provide further input. Hello again, I am the me too-guy with console-access. I am not a programmer and it is the first time I see debugging screen. It deadlocked again, and I did as advised above: (ddb: show lockedvnods; ps ; panic) but did not understand much of the output. Looked maybe like syncer and swap_pager was locked? Do i need to write all this down or can I get the output saved somewhere? I got a 32MB coredump but the same lack of understanding applies. Please tell me if I can be of any help! This is fun. Do you have the ability to connect another computer by RS-232? It's easy to get a serial terminal console going (err that is if you find the right guide as opposed to stabbing blindly and just referencing man pages as I like to do.) The coredump should supply the same (and more) information, and someone can walk through with you doing a post-mortem gdb session. For example, try doing the following now that you have the coredump: # ps wwwauxlH -N /boot/kernel/kernel -M /var/crash/vmcore.whatever Sure, I will get a serial terminal console going and try to repeat this process from it. In the meantime here is output from the above ps command provided as attachement. -- Lars -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND UID PPID CPU PRI NI MWCHAN root0 0.0 0.0 0 0 ?? DLs 1Jan70 0:00.09 [swapper] 0 0 0 -16 0 vmwait root1 0.0 0.0 748 0 ?? DLs 1Jan70 0:00.55 [init] 0 0 52 8 0 wait root2 0.0 0.0 0 0 ?? DL1Jan70 0:02.31 [g_event] 0 0 0 -8 0 - root3 0.0 0.0 0 0 ?? DL1Jan70 0:10.52 [g_up] 0 0 0 -8 0 - root4 0.0 0.0 0 0 ?? DL1Jan70 0:09.21 [g_down] 0 0 0 -8 0 - root5 0.0 0.0 0 0 ?? DL1Jan70 0:00.00 [kqueue taskq] 0 0 0 8 0 - root6 0.0 0.0 0 0 ?? DL1Jan70 0:00.00 [thread taskq] 0 0 0 8 0 - root7 0.0 0.0 0 0 ?? DL1Jan70 0:01.71 [fdc0] 0 0 0 -8 0 - root8 0.0 0.0 0 0 ?? DL1Jan70 0:13.57 [pagedaemon] 0 0 1 -16 0 wswbuf root9 0.0 0.0 0 0 ?? DL1Jan70 0:02.10 [vmdaemon] 0 0 0 20 0 psleep root 10 0.0 0.0 0 0 ?? DL1Jan70 0:00.00 [ktrace] 0 0 0 -16 0 ktrace root 11 0.0 0.0 0 0 ?? RL1Jan70 44:26.68 [idle] 0 0 67 171 0 - root 12 0.0 0.0 0 0 ?? WL1Jan70 0:00.00 [irq0: clk] 0 0 0 -84 0 - root 13 0.0 0.0 0 0 ?? RL1Jan70 0:00.02 [irq1: atkbd0] 0 0 1 -60 0 - root 14 0.0 0.0 0 0 ?? WL1Jan70 0:00.00 [irq3:] 0 0 0 -21 0 - root 15 0.0
Re: Swapfile problem in 6?
On Wed, 2005-Nov-16 04:21:09 -0800, Rob wrote: If not, then what should I remove/keep from the above list, to allow the deadlock to reappear and still be able to debug the problem? The minimum you need to get into DDB and use GDB off-line is makeoptions DEBUG=-g options KDB options DDB options BREAK_TO_DEBUGGER -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Wed, Nov 16, 2005 at 04:01:36PM +0100, Lars Kristiansen wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. First question is: Does the system still deadlock? INVARIANTS and WITNESS will have added sanity checks which might have picked up the problem. 1) Can I debug a kernel that does not crash, but just hangs in a deadlock? Everything seems to be frozen, except pinging the PC Have a look at http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-online-ddb.html and ddb(4). Unless you have another system handy, you might like to print out ddb(4) - it's difficult to read man pages when you're in the kernel debugger :-). 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters - which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot - see http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-gdb.html If in doubt, post the output from the above commands here and someone will hopefully provide further input. Hello again, I am the me too-guy with console-access. I am not a programmer and it is the first time I see debugging screen. It deadlocked again, and I did as advised above: (ddb: show lockedvnods; ps ; panic) but did not understand much of the output. Looked maybe like syncer and swap_pager was locked? Do i need to write all this down or can I get the output saved somewhere? I got a 32MB coredump but the same lack of understanding applies. Please tell me if I can be of any help! This is fun. Do you have the ability to connect another computer by RS-232? It's easy to get a serial terminal console going (err that is if you find the right guide as opposed to stabbing blindly and just referencing man pages as I like to do.) The coredump should supply the same (and more) information, and someone can walk through with you doing a post-mortem gdb session. For example, try doing the following now that you have the coredump: # ps wwwauxlH -N /boot/kernel/kernel -M /var/crash/vmcore.whatever Sure, I will get a serial terminal console going and try to repeat this process from it. In the meantime here is output from the above ps command provided as attachement. -- Lars Yes, it deadlocked almost immediately. A debug session is attached. -- Lars -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] KDB: enter: Line break on console [thread pid 11 tid 15 ] Stopped at 0xc0671fcb = kdb_enter+0x2b:nop db show lockedvnods Locked vnodes 0xc12cdbb0: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL (count 1) by thread 0xc1143a80 (pid 40) 0xc12cdaa0: tag ufs, type VREG usecount 1, writecount 0, refcount 3 mountedhere 0
Re: Swapfile problem in 6?
--- Peter Jeremy [EMAIL PROTECTED] wrote: Since your /home is almost empty, how about (temporarily) moving the contents into /usr and swapping onto ad0s1e rather than into a swapfile. This should at least enable you to build a debug kernel. I'm now indeed building a new kernel, without using a swapfile. I have following in the kernel config: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? First I used 128 MB swapfile on root partition; then tried again with a 128 MB swapfile on /var. However, exactly the same deadlock occurs: Have you pre-allocated the swapfile or is it being allocated as necessary? If the latter, try dd if=/dev/zero of=swapfile bs=1m count=128 Yes, I always 'zero' the swapfile this way, before adding. Because I have used this swapfile construct extensively with previous 5-Stable on the same PC, I assume that harddisk and hardware are OK. Therefore my conclusion is, that there's some problem related to swapfile with 6. I will try to debug as much as I can, to find out why the machine becomes dead at a particular point while using a swapfile (fortunately this is very reproducible). I have never debugged a kernel, so I may ask soon for some assistance. Already now I wonder: 1) Can I debug a kernel that does not crash, but just hangs in a deadlock? Everything seems to be frozen, except pinging the PC 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Thanks, Rob. __ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
One more thing, for those who are reading this thread. Concerning my kernel locking up while using a swapfile: I had an infinite while loop running during the kernel compilation, printing out 'swapinfo' every 15 seconds. The last output before the machine goes dead is this: /dev/ad0s1b 39848118203984830% /dev/md0 131072 5908 131072 5% Total 17092017728 15319210% which means that swap space is OK right before the freeze, but within 15 seconds from here, the kernel locks up Rob. __ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. First question is: Does the system still deadlock? INVARIANTS and WITNESS will have added sanity checks which might have picked up the problem. 1) Can I debug a kernel that does not crash, but just hangs in a deadlock? Everything seems to be frozen, except pinging the PC Have a look at http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-online-ddb.html and ddb(4). Unless you have another system handy, you might like to print out ddb(4) - it's difficult to read man pages when you're in the kernel debugger :-). 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. Basically, wait until your system deadlocks. BREAK into DDB. As a start, run 'show lockedvnods', 'ps'. My guess is that you'll see a lock that has a number of waiters - which is probably the culprit. Use 'panic' to get a crashdump and then you can use kgdb to rummage around once you reboot - see http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-gdb.html If in doubt, post the output from the above commands here and someone will hopefully provide further input. -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
Rob [EMAIL PROTECTED] wrote: Kris Kennaway [EMAIL PROTECTED] wrote: Since you can compile a kernel without it, add DDB, WITNESS and INVARIANTS support, then trigger the deadlock with the swapfile, break to DDB and examine the state of the machine. See the chapter on kernel debugging in the developers handbook for more instructions. Thanks, but for now, I cannot compile a new kernel, because the kernel compilation terminates with insufficient swap space error. Apparently 32 MB is not enough for a new kernel compilation. Remove the -pipe option from CFLAGS and COPTFLAGS in your /etc/make.conf file. That will serialize compiler stages that normally run in parallel, so it'll require less memory. Also make sure that no processes run that you don't need, i.e. kill cron, usbd, sendmail, ntpd and whatever else you don't absolutely need. I don't know if those measures will enable you to compile the kernel with 32 Mbytes, but it's worth a try. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. C is quirky, flawed, and an enormous success. -- Dennis M. Ritchie. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
--- Peter Jeremy [EMAIL PROTECTED] wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. If I merge all these options into the GENERIC kernel config file, the kernel compilation ends with error: $ make buildkernel KERNCONF=DEBUGKERNEL [...snip...] cc -c -O -pipe ... /usr/src/sys/dev/ata/atapi-cd.c /usr/src/sys/dev/ata/atapi-cd.c: In function `acd_geom_attach': /usr/src/sys/dev/ata/atapi-cd.c:181: warning: implicit declaration of function `_sx_assert' /usr/src/sys/dev/ata/atapi-cd.c:181: warning: nested extern declaration of `_sx_assert' *** Error code 1 Stop in /usr/obj/usr/src/sys/MYKERNEL. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. -- In /etc/make.conf, I have: CFLAGS= -O -pipe NO_PROFILE=true NO_PF=true RUBY_VER=1.8 PERL_VER=5.8.7 PERL_VERSION=5.8.7 Hence, so far no luck with building a debug kernel with 6. Any ideas what's wrong here? Thanks, Rob. __ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 07:16:47AM -0800, Rob wrote: --- Peter Jeremy [EMAIL PROTECTED] wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. If I merge all these options into the GENERIC kernel config file, the kernel compilation ends with error: $ make buildkernel KERNCONF=DEBUGKERNEL [...snip...] cc -c -O -pipe ... /usr/src/sys/dev/ata/atapi-cd.c /usr/src/sys/dev/ata/atapi-cd.c: In function `acd_geom_attach': /usr/src/sys/dev/ata/atapi-cd.c:181: warning: implicit declaration of function `_sx_assert' /usr/src/sys/dev/ata/atapi-cd.c:181: warning: nested extern declaration of `_sx_assert' *** Error code 1 Stop in /usr/obj/usr/src/sys/MYKERNEL. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. You may not have INVARIANT_SUPPORT -- I don't see that listed explicitly, but it's necessary as well. -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 02:15:37AM -0800, Rob wrote: One more thing, for those who are reading this thread. Concerning my kernel locking up while using a swapfile: I had an infinite while loop running during the kernel compilation, printing out 'swapinfo' every 15 seconds. The last output before the machine goes dead is this: /dev/ad0s1b 39848118203984830% /dev/md0 131072 5908 131072 5% Total 17092017728 15319210% which means that swap space is OK right before the freeze, but within 15 seconds from here, the kernel locks up In my experience, fifteen seconds is enough to fill up 130+MB of swap, so a much smaller polling interval is necessary. -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 07:16:47AM -0800, Rob wrote: --- Peter Jeremy [EMAIL PROTECTED] wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: makeoptions DEBUG=-g options INVARIANTS options WITNESS options WITNESS_KDB options KDB options DDB options DDB_NUMSYM options GDB Is that enough? If your system is headless, you probably want 'options BREAK_TO_DEBUGGER' as well. If I merge all these options into the GENERIC kernel config file, the kernel compilation ends with error: $ make buildkernel KERNCONF=DEBUGKERNEL [...snip...] cc -c -O -pipe ... /usr/src/sys/dev/ata/atapi-cd.c /usr/src/sys/dev/ata/atapi-cd.c: In function `acd_geom_attach': /usr/src/sys/dev/ata/atapi-cd.c:181: warning: implicit declaration of function `_sx_assert' /usr/src/sys/dev/ata/atapi-cd.c:181: warning: nested extern declaration of `_sx_assert' *** Error code 1 Stop in /usr/obj/usr/src/sys/MYKERNEL. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. You may not have INVARIANT_SUPPORT -- I don't see that listed explicitly, but it's necessary as well. That seemed to get the buildkernel going for me at least. I have been following this thread and can confirm this freezing when using swapfile. It happens here while rebuilding different ports, after upgrade to 6.0-release. A couple of the times but not everytime the following message was repeatedly displayed at the console before freezing: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 273427, size: 4096 To mostly rule out diskfailure I also disabled this swapfile and tried to have the swapfile on another disk. Additional information about frozen behavior: When frozen I can change virtual terminals: alt-F2, alt-F3... But that is all i can do whith keyboard. ie. I cannot write username. Got a swap partition on another disk now. That speeds things up too. However, this is a pentium2-200MHz so I cannot tell if buildkernel was a success until tomorrow. Never done debugging before, but I will continue to follow this thread tomorrow and try to do what is adviced here if the cause has not been identified in the meantime. Regards -- Lars -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 02:15:37AM -0800, Rob wrote: One more thing, for those who are reading this thread. Concerning my kernel locking up while using a swapfile: I had an infinite while loop running during the kernel compilation, printing out 'swapinfo' every 15 seconds. The last output before the machine goes dead is this: /dev/ad0s1b 39848118203984830% /dev/md0 131072 5908 131072 5% Total 17092017728 15319210% which means that swap space is OK right before the freeze, but within 15 seconds from here, the kernel locks up In my experience, fifteen seconds is enough to fill up 130+MB of swap, so a much smaller polling interval is necessary. Probably true, but when trying with only one swap partition instead of swapfile, and that partition gets filled up, the build process failed and the computer continued to work as normal. Regards -- Lars -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 09:49:57PM +0100, Lars Kristiansen wrote: On Tue, Nov 15, 2005 at 02:15:37AM -0800, Rob wrote: One more thing, for those who are reading this thread. Concerning my kernel locking up while using a swapfile: I had an infinite while loop running during the kernel compilation, printing out 'swapinfo' every 15 seconds. The last output before the machine goes dead is this: /dev/ad0s1b 39848118203984830% /dev/md0 131072 5908 131072 5% Total 17092017728 15319210% which means that swap space is OK right before the freeze, but within 15 seconds from here, the kernel locks up In my experience, fifteen seconds is enough to fill up 130+MB of swap, so a much smaller polling interval is necessary. Probably true, but when trying with only one swap partition instead of swapfile, and that partition gets filled up, the build process failed and the computer continued to work as normal. Yeah, it sounds like you're running into a buffer cache/VFS/VM deadlock. They're nasty, but generally not hard to fix individually. -- Brian Fundakowski Feldman \'[ FreeBSD ]''\ [EMAIL PROTECTED] \ The Power to Serve! \ Opinions expressed are my own. \,,\ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 09:44:12PM +0100, Lars Kristiansen wrote: A couple of the times but not everytime the following message was repeatedly displayed at the console before freezing: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 273427, size: 4096 AFAICT that is just a trigger-happy timer..it's supposed to detect when a swap operation took too long to complete, but it also triggers on swapfiles since they're so much less efficient (i.e. slower) than swapping onto a bare device. Kris pgpzur3FSifHZ.pgp Description: PGP signature
Re: Swapfile problem in 6?
On 11/16/05 07:49, Lars Kristiansen wrote: I had an infinite while loop running during the kernel compilation, printing out 'swapinfo' every 15 seconds. The last output before the machine goes dead is this: /dev/ad0s1b 39848118203984830% /dev/md0 131072 5908 131072 5% Total 17092017728 15319210% Quite possibly not related, but on 5.4 I can reliably cause a panic by attaching md0 to a file residing on an NFS drive and then use it for swap. After 3-5k of used swap on md0, the box dies and reboots. If I run swapon directly on the NFS file, things work fine, but it did mean that I couldn't use the default addswap rc script to set up the swap on my diskless system. I'm not sure whether it's supposed to work or not, but I haven't had time to look into it yet. Just thought it might be another datapoint for someone in the know to consider, just in case it is related. Cheers, /Johny -- Johny Mattsson - Making IT work ,-. ,-. ,-. When all else fails, http://www.earthmagic.org _.' `-' `-' Murphy's Law still works! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Wed, Nov 16, 2005 at 09:26:43AM +1100, Johny Mattsson wrote: On 11/16/05 07:49, Lars Kristiansen wrote: I had an infinite while loop running during the kernel compilation, printing out 'swapinfo' every 15 seconds. The last output before the machine goes dead is this: /dev/ad0s1b 39848118203984830% /dev/md0 131072 5908 131072 5% Total 17092017728 15319210% Quite possibly not related, but on 5.4 I can reliably cause a panic by attaching md0 to a file residing on an NFS drive and then use it for swap. After 3-5k of used swap on md0, the box dies and reboots. If I run swapon directly on the NFS file, things work fine, but it did mean that I couldn't use the default addswap rc script to set up the swap on my diskless system. I'm not sure whether it's supposed to work or not, but I haven't had time to look into it yet. Just thought it might be another datapoint for someone in the know to consider, just in case it is related. It's a situation few people have probably tried. If it persists on 6.0, please go through the same reporting procedure (debugging traceback, file PR). Kris pgpk2nma0h72W.pgp Description: PGP signature
Re: Swapfile problem in 6?
--- Peter Jeremy [EMAIL PROTECTED] wrote: On Tue, 2005-Nov-15 02:08:12 -0800, Rob wrote: 2) Is such debugging possible on a headless PC without a keyboard attached? I do have serial console access. Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. What is this serial BREAK? How do I send a serial BREAK at the serial console? Is this some magic key combination? And, eh, at the moment of deadlock, there is no response at all from the serial console; will the serial BREAK not be bothered by that? Rob. __ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 06:09:33PM -0800, Rob wrote: ... Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. What is this serial BREAK? How do I send a serial BREAK at the serial console? Is this some magic key combination? I'm probably saying something about my age by doing this :-} A BREAK (in serial communications) is an absence of start or stop bits for more than a character's worth of bits, is handled as a framing error, and is distinct from any character. Now, to generate one, you would normally hit the BREAK key of your terminal (Control-BREAK on some; I think Hazeltine had this feature of dubious merit). If you are using a serial communications program (such as tip) instead of an ASCII terminal, it depends on the program you're using. In the case of tip, the sequence ~# at the beginning of a line will generate tip's best approximation of a framing error. In the case of kermit, Control-\B does it. And, eh, at the moment of deadlock, there is no response at all from the serial console; will the serial BREAK not be bothered by that? The serial BREAK won't be bothered at all. :-} The issue is (if I understand properly) whether or not there's enough of the system to be able to have the tty driver recognize the framing error and do something appropriate as a result. Peace, david -- David H. Wolfskill [EMAIL PROTECTED] Prediction is difficult, especially if it involves the future. -- Niels Bohr See http://www.catwhisker.org/~david/publickey.gpg for public key. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 07:18:35PM -0800, David Wolfskill wrote: On Tue, Nov 15, 2005 at 06:09:33PM -0800, Rob wrote: ... Yes. See above URL. The advantage is that you can (hopefully) capture a log of your debug session. Send a serial BREAK and you should get a DDB prompt. What is this serial BREAK? How do I send a serial BREAK at the serial console? Is this some magic key combination? I'm probably saying something about my age by doing this :-} A BREAK (in serial communications) is an absence of start or stop bits for more than a character's worth of bits, is handled as a framing error, and is distinct from any character. If your serial program has trouble sending a break, you can use the ALT_BREAK_TO_DEBUGGER config option which instead listens for a sequence of ascii input: #define KEY_CR 13 /* CR '\r' */ #define KEY_TILDE 126 /* ~ */ #define KEY_CRTLB 2 /* ^B */ Kris pgpg8eJLqZ8qu.pgp Description: PGP signature
Re: Swapfile problem in 6?
Kris Kennaway wrote this message on Tue, Nov 15, 2005 at 22:35 -0500: What is this serial BREAK? How do I send a serial BREAK at the serial console? Is this some magic key combination? I'm probably saying something about my age by doing this :-} A BREAK (in serial communications) is an absence of start or stop bits for more than a character's worth of bits, is handled as a framing error, and is distinct from any character. If your serial program has trouble sending a break, you can use the ALT_BREAK_TO_DEBUGGER config option which instead listens for a sequence of ascii input: #define KEY_CR 13 /* CR '\r' */ #define KEY_TILDE 126 /* ~ */ #define KEY_CRTLB 2 /* ^B */ ALT_BREAK_TO_DEBUGGER has an advantage over break is that some terminal servers and computers when you turn them off (or power is disconnected) will send a BREAK down the line, which can cause your machine to stop when you didn't want to.. -- John-Mark Gurney Voice: +1 415 225 5579 All that I will do, has been done, All that I have, has not. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6?
On Tue, Nov 15, 2005 at 07:18:35PM -0800, David Wolfskill wrote: On Tue, Nov 15, 2005 at 06:09:33PM -0800, Rob wrote: ... What is this serial BREAK? How do I send a serial BREAK at the serial console? Is this some magic key combination? I'm probably saying something about my age by doing this :-} A BREAK (in serial communications) is an absence of start or stop bits for more than a character's worth of bits, is handled as a framing error, and is distinct from any character. Now, to generate one, you would normally hit the BREAK key of your terminal (Control-BREAK on some; I think Hazeltine had this feature of dubious merit). If you are using a serial communications program (such as tip) instead of an ASCII terminal, it depends on the program you're using. In the case of tip, the sequence ~# at the beginning of a line will generate tip's best approximation of a framing error. In the case of kermit, Control-\B does it. If you're using a terminal program on something like Windows or a Macintosh, there's usually a menu option to send a BREAK. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Swapfile problem in 6? (was: 6.0: during kernel compilation, 'kernel linking' freezes PC)
--- Rob [EMAIL PROTECTED] wrote: Hi, I have upgraded a Pentium-1 PC from 5 to 6-Stable, which went smoothly. Now I'm running 6 with GENERIC kernel, and I want to compile a new kernel. When I start a new kernel compilation, this process *always* freezes the PC (no crash) when reaching the point of linking kernel; this is 100 percent reproducible!! Then all communication with the PC is frozen; no serial port response, no ssh response etc. However, I can ping the PC. Only power off/on brings the PC back to live. Any idea how I can further investigate. /var/log/messages has no special information while the freezing occurred. The output of 'dmesg' of this PC is here: http://surfion.snu.ac.kr/~lahaye/dmesg Problem kind of solved: As this PC has only 32 MB Ram, I add a swapfile. However, when I compile the new kernel without the swapfile, all goes well. Since the freeze is very reproducible with the swapfile (always at the 'kernel linking' stage), and immediately disappears when I do not use the swapfile, I think there's something wrong with using swapfile in 6.0. Is it? I do this: dd if=/dev/zero of=/swapfile bs=1024k count=128 chmod 0600 /swapfile mdconfig -a -t vnode -f /swapfile -u 0 swapon /dev/md0 Is this OK? Rob. __ Start your day with Yahoo! - Make it your home page! http://www.yahoo.com/r/hs ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6? (was: 6.0: during kernel compilation, 'kernel linking' freezes PC)
On Mon, Nov 14, 2005 at 05:52:42PM -0800, Rob wrote: --- Rob [EMAIL PROTECTED] wrote: Hi, I have upgraded a Pentium-1 PC from 5 to 6-Stable, which went smoothly. Now I'm running 6 with GENERIC kernel, and I want to compile a new kernel. When I start a new kernel compilation, this process *always* freezes the PC (no crash) when reaching the point of linking kernel; this is 100 percent reproducible!! Then all communication with the PC is frozen; no serial port response, no ssh response etc. However, I can ping the PC. Only power off/on brings the PC back to live. Any idea how I can further investigate. /var/log/messages has no special information while the freezing occurred. The output of 'dmesg' of this PC is here: http://surfion.snu.ac.kr/~lahaye/dmesg Problem kind of solved: As this PC has only 32 MB Ram, I add a swapfile. However, when I compile the new kernel without the swapfile, all goes well. Since the freeze is very reproducible with the swapfile (always at the 'kernel linking' stage), and immediately disappears when I do not use the swapfile, I think there's something wrong with using swapfile in 6.0. Is it? I do this: dd if=/dev/zero of=/swapfile bs=1024k count=128 chmod 0600 /swapfile mdconfig -a -t vnode -f /swapfile -u 0 swapon /dev/md0 Is this OK? Looks fine, and I use this myself (on a much larger scale). Are you sure it's freezing and not just taking a really long time? Whenever your system is swapping performance will be terrible, and that goes at least double for swapping onto a file and not a device. What happens if you leave it for an hour or two (or overnight)? Kris pgptYcxFB6Uv8.pgp Description: PGP signature
Re: Swapfile problem in 6? (was: 6.0: during kernel compilation, 'kernel linking' freezes PC)
--- Kris Kennaway [EMAIL PROTECTED] wrote: On Mon, Nov 14, 2005 at 05:52:42PM -0800, Rob wrote: Problem kind of solved: As this PC has only 32 MB Ram, I add a swapfile. However, when I compile the new kernel without the swapfile, all goes well. Since the freeze is very reproducible with the swapfile (always at the 'kernel linking' stage), and immediately disappears when I do not use the swapfile, I think there's something wrong with using swapfile in 6.0. Is it? I do this: dd if=/dev/zero of=/swapfile bs=1024k count=128 chmod 0600 /swapfile mdconfig -a -t vnode -f /swapfile -u 0 swapon /dev/md0 Is this OK? Looks fine, and I use this myself (on a much larger scale). Are you sure it's freezing and not just taking a really long time? Whenever your system is swapping performance will be terrible, and that goes at least double for swapping onto a file and not a device. What happens if you leave it for an hour or two (or overnight)? I left it overnight twice for performing this 'kernel linking' stage. In both cases the machine remained dead. Without the swapfile, this 'kernel linking' stage completes in just about less than a minute. So I think it's not my impatience, but there is another problem with my swapfile or with 6 :). Rob. __ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6? (was: 6.0: during kernel compilation, 'kernel linking' freezes PC)
On Mon, Nov 14, 2005 at 06:08:27PM -0800, Rob wrote: I left it overnight twice for performing this 'kernel linking' stage. In both cases the machine remained dead. Without the swapfile, this 'kernel linking' stage completes in just about less than a minute. So I think it's not my impatience, but there is another problem with my swapfile or with 6 :). Since you can compile a kernel without it, add DDB, WITNESS and INVARIANTS support, then trigger the deadlock with the swapfile, break to DDB and examine the state of the machine. See the chapter on kernel debugging in the developers handbook for more instructions. Kris pgpJ6WLBy6qWO.pgp Description: PGP signature
Re: Swapfile problem in 6? (was: 6.0: during kernel compilation, 'kernel linking' freezes PC)
--- Kris Kennaway [EMAIL PROTECTED] wrote: On Mon, Nov 14, 2005 at 06:08:27PM -0800, Rob wrote: I left it overnight twice for performing this 'kernel linking' stage. In both cases the machine remained dead. Without the swapfile, this 'kernel linking' stage completes in just about less than a minute. So I think it's not my impatience, but there is another problem with my swapfile or with 6 :). Since you can compile a kernel without it, add DDB, WITNESS and INVARIANTS support, then trigger the deadlock with the swapfile, break to DDB and examine the state of the machine. See the chapter on kernel debugging in the developers handbook for more instructions. Thanks, but for now, I cannot compile a new kernel, because the kernel compilation terminates with insufficient swap space error. Apparently 32 MB is not enough for a new kernel compilation. So for now I'm stuck with the GENERIC kernel. This is my partitioning: /dev/ad0s1a253678 34446 19893815%/ /dev/ad0s1b 39848 8168 3984820%(swap) /dev/ad0s1d253678 152958 8042666%/var /dev/ad0s1e253678 6016 227368 3%/home /dev/ad0s1f 1624576 727274 76733649%/usr First I used 128 MB swapfile on root partition; then tried again with a 128 MB swapfile on /var. However, exactly the same deadlock occurs: when the kernel compilation reaches 'linking kernel' the PC is dead (no crash). Although I can ping the PC, there's no response to any other service; also serial port console is dead. Without using the swapfile, the compilation easily passes through the 'linking kernel' stage, but somewhat later terminates as swap space is insufficient. With 5-Stable, I used the swapfile extensively for recompiling new kernels and worlds, without any problem. I only can conclude that something is wrong with the swapfile construct in 6. Is 128 MB too big for a swapfile in 6? I'll try again with a smaller swapfile... Any other ideas? dmesg output of this PC is here: http://surfion.snu.ac.kr/~lahaye/dmesg Thanks, Rob. __ Yahoo! FareChase: Search multiple travel sites in one click. http://farechase.yahoo.com ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Swapfile problem in 6? (was: 6.0: during kernel compilation, 'kernel linking' freezes PC)
On Mon, 2005-Nov-14 22:38:59 -0800, Rob wrote: --- Kris Kennaway [EMAIL PROTECTED] wrote: Since you can compile a kernel without it, add DDB, WITNESS and INVARIANTS support, then trigger the deadlock with the swapfile, break to DDB and examine the state of the machine. See the chapter on kernel debugging in the developers handbook for more instructions. Thanks, but for now, I cannot compile a new kernel, because the kernel compilation terminates with insufficient swap space error. Unfortunately, we're probably not going to be able to provide much assistance without knowing more about what is happening when it deadlocks. (As Kris requests). BTW, you should probably make sure you keep makeoptions DEBUG=-g and set dumpdev in rc.conf (which will make it possible to capture and use a crashdump if you can trigger one). Apparently 32 MB is not enough for a new kernel compilation. Quite probably. This is my partitioning: /dev/ad0s1a253678 34446 19893815%/ /dev/ad0s1b 39848 8168 3984820%(swap) /dev/ad0s1d253678 152958 8042666%/var /dev/ad0s1e253678 6016 227368 3%/home /dev/ad0s1f 1624576 727274 76733649%/usr Since your /home is almost empty, how about (temporarily) moving the contents into /usr and swapping onto ad0s1e rather than into a swapfile. This should at least enable you to build a debug kernel. First I used 128 MB swapfile on root partition; then tried again with a 128 MB swapfile on /var. However, exactly the same deadlock occurs: Have you pre-allocated the swapfile or is it being allocated as necessary? If the latter, try dd if=/dev/zero of=swapfile bs=1m count=128 -- Peter Jeremy ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
