Re: ffs snapshot lockup

2006-10-06 Thread Kostik Belousov
On Thu, Oct 05, 2006 at 10:01:07AM -0400, Vivek Khera wrote: On Oct 5, 2006, at 4:30 AM, Kostik Belousov wrote: The network load was minimal at the time. I had everyone log out and close mail etc. What were the symptoms of locked system ? Could you log in on console, or do

Re: ffs snapshot lockup

2006-10-06 Thread Kris Kennaway
On Fri, Oct 06, 2006 at 10:39:50AM +0300, Kostik Belousov wrote: On Thu, Oct 05, 2006 at 10:01:07AM -0400, Vivek Khera wrote: On Oct 5, 2006, at 4:30 AM, Kostik Belousov wrote: The network load was minimal at the time. I had everyone log out and close mail etc. What were

Re: ffs snapshot lockup

2006-10-06 Thread Kostik Belousov
On Fri, Oct 06, 2006 at 02:11:05PM -0400, Vivek Khera wrote: On Oct 6, 2006, at 1:57 PM, Kris Kennaway wrote: This is very strange. You 3 instances of getty where just reading the tty input, and all suspectible processes (like sshd) are waiting on net events. No processes are blocked on

Re: ffs snapshot lockup

2006-10-06 Thread Kris Kennaway
On Fri, Oct 06, 2006 at 02:11:05PM -0400, Vivek Khera wrote: On Oct 6, 2006, at 1:57 PM, Kris Kennaway wrote: This is very strange. You 3 instances of getty where just reading the tty input, and all suspectible processes (like sshd) are waiting on net events. No processes are blocked on

Re: ffs snapshot lockup

2006-10-06 Thread Vivek Khera
On Oct 6, 2006, at 2:20 PM, Kostik Belousov wrote: If it does lock up again, I'll be sure to let you know! Was this system patched by the stuff I submitted to you ? yes. i did not update anything except adding the xl driver to the kernel, so as to minimize changes. if this holds

Re: ffs snapshot lockup

2006-10-06 Thread Vivek Khera
On Oct 6, 2006, at 2:31 PM, Kris Kennaway wrote: I'm now convinced it was all stemming from some bug in bge driver (at least for my specific chipset.) Last night I put in an old spare 3c905 NIC and turned off the motherboard bge via BIOS. We'd be interested in diagnosing this problem

Re: ffs snapshot lockup

2006-10-05 Thread Kostik Belousov
On Wed, Oct 04, 2006 at 05:16:53PM -0400, Vivek Khera wrote: On Oct 3, 2006, at 4:43 AM, Kostik Belousov wrote: Details are posted at http://vivek.khera.org/scratch/crashlogs/ I have the crashdumps available to a kernel hacker upon request (i'd rather not make them generally available to

Re: ffs snapshot lockup

2006-10-05 Thread Vivek Khera
On Oct 5, 2006, at 4:30 AM, Kostik Belousov wrote: The network load was minimal at the time. I had everyone log out and close mail etc. What were the symptoms of locked system ? Could you log in on console, or do something at the shell prompt on console ? Console was non-responsive.

Re: ffs snapshot lockup

2006-10-04 Thread Vivek Khera
On Oct 3, 2006, at 4:43 AM, Kostik Belousov wrote: Details are posted at http://vivek.khera.org/scratch/crashlogs/ I have the crashdumps available to a kernel hacker upon request (i'd rather not make them generally available to the public...) It seems that you have snapshotted fs exported by

Re: ffs snapshot lockup

2006-10-04 Thread Kostik Belousov
On Wed, Oct 04, 2006 at 09:56:54AM -0400, Vivek Khera wrote: On Oct 3, 2006, at 4:43 AM, Kostik Belousov wrote: Details are posted at http://vivek.khera.org/scratch/crashlogs/ I have the crashdumps available to a kernel hacker upon request (i'd rather not make them generally available to

Re: ffs snapshot lockup

2006-10-04 Thread Kris Kennaway
On Wed, Oct 04, 2006 at 05:08:08PM +0300, Kostik Belousov wrote: On Wed, Oct 04, 2006 at 09:56:54AM -0400, Vivek Khera wrote: On Oct 3, 2006, at 4:43 AM, Kostik Belousov wrote: Details are posted at http://vivek.khera.org/scratch/crashlogs/ I have the crashdumps available to a

Re: ffs snapshot lockup

2006-10-04 Thread Vivek Khera
On Oct 4, 2006, at 12:39 PM, Kris Kennaway wrote: The only thing I think was running at the time would be a large file copy from a remote system to this one using rsync. As I understand, you got the panic. Then, you shall post the panic message. If you have core file, then running kgdb on

Re: ffs snapshot lockup

2006-10-04 Thread Kostik Belousov
On Wed, Oct 04, 2006 at 01:06:37PM -0400, Vivek Khera wrote: On Oct 4, 2006, at 12:39 PM, Kris Kennaway wrote: The only thing I think was running at the time would be a large file copy from a remote system to this one using rsync. As I understand, you got the panic. Then, you shall

Re: ffs snapshot lockup

2006-10-04 Thread Vivek Khera
On Oct 4, 2006, at 1:16 PM, Kostik Belousov wrote: #9 0x80369166 in Xtimerint () at apic_vector.S:153 #10 0x801d1c05 in bge_intr (xsc=0x8698e010) at bus.h:241 Did you have any problems with your network card ?

Re: ffs snapshot lockup

2006-10-04 Thread Vivek Khera
On Oct 4, 2006, at 1:16 PM, Kostik Belousov wrote: #10 0x801d1c05 in bge_intr (xsc=0x8698e010) at bus.h:241 Did you have any problems with your network card ? This seems to be quite popular plot in recent times. Another

Re: ffs snapshot lockup

2006-10-04 Thread Kris Kennaway
On Wed, Oct 04, 2006 at 01:06:37PM -0400, Vivek Khera wrote: On Oct 4, 2006, at 12:39 PM, Kris Kennaway wrote: The only thing I think was running at the time would be a large file copy from a remote system to this one using rsync. As I understand, you got the panic. Then, you shall

Re: ffs snapshot lockup

2006-10-04 Thread Vivek Khera
On Oct 4, 2006, at 3:41 PM, Kris Kennaway wrote: from what i read in the output from kgdb, it seems that something locked the kernel and we broke to debugger from the watchdog timeout (I enable software watchdog). Hmm, be careful with that - if you set the timeout too low (and note that for

Re: ffs snapshot lockup

2006-10-04 Thread Kris Kennaway
On Wed, Oct 04, 2006 at 03:53:54PM -0400, Vivek Khera wrote: On Oct 4, 2006, at 3:41 PM, Kris Kennaway wrote: from what i read in the output from kgdb, it seems that something locked the kernel and we broke to debugger from the watchdog timeout (I enable software watchdog). Hmm, be

Re: ffs snapshot lockup

2006-10-04 Thread Vivek Khera
On Oct 3, 2006, at 4:43 AM, Kostik Belousov wrote: Details are posted at http://vivek.khera.org/scratch/crashlogs/ I have the crashdumps available to a kernel hacker upon request (i'd rather not make them generally available to the public...) It seems that you have snapshotted fs exported by

Re: ffs snapshot lockup

2006-10-04 Thread Peter Holm
On Wed, Oct 04, 2006 at 03:41:48PM -0400, Kris Kennaway wrote: On Wed, Oct 04, 2006 at 01:06:37PM -0400, Vivek Khera wrote: On Oct 4, 2006, at 12:39 PM, Kris Kennaway wrote: The only thing I think was running at the time would be a large file copy from a remote system to this one

Re: ffs snapshot lockup

2006-10-03 Thread Kostik Belousov
On Mon, Oct 02, 2006 at 03:23:49PM -0400, Vivek Khera wrote: On Sep 22, 2006, at 4:36 PM, Kris Kennaway wrote: Start by enabling INVARIANTS, INVARIANT_SUPPORT, DEBUG_LOCKS and DEBUG_VFS_LOCKS, then run 'show lockedvnods' and 'alltrace' in DDB (spammy, need that serial console), or at least

Re: ffs snapshot lockup

2006-10-03 Thread Vivek Khera
On Oct 3, 2006, at 4:43 AM, Kostik Belousov wrote: Details are posted at http://vivek.khera.org/scratch/crashlogs/ I have the crashdumps available to a kernel hacker upon request (i'd rather not make them generally available to the public...) It seems that you have snapshotted fs exported by

Re: ffs snapshot lockup

2006-10-02 Thread Vivek Khera
On Sep 22, 2006, at 4:36 PM, Kris Kennaway wrote: Start by enabling INVARIANTS, INVARIANT_SUPPORT, DEBUG_LOCKS and DEBUG_VFS_LOCKS, then run 'show lockedvnods' and 'alltrace' in DDB (spammy, need that serial console), or at least trace the running processes (show allpcpu) and those listed in

Re: ffs snapshot lockup

2006-09-26 Thread Vivek Khera
On Sep 25, 2006, at 2:58 PM, Jeremy Chadwick wrote: This problem sounds awfully like what I reported in a recent PR. Now I'm starting to have my doubts as to whether or not my hardware was indeed at fault... http://www.freebsd.org/cgi/query-pr.cgi?pr=103435 I think my issue is a bit

Re: ffs snapshot lockup

2006-09-25 Thread Vivek Khera
On Sep 22, 2006, at 4:36 PM, Kris Kennaway wrote: Start by enabling INVARIANTS, INVARIANT_SUPPORT, DEBUG_LOCKS and DEBUG_VFS_LOCKS, then run 'show lockedvnods' and 'alltrace' in DDB (spammy, need that serial console), or at least trace the running processes (show allpcpu) and those listed in

Re: ffs snapshot lockup

2006-09-25 Thread Jeremy Chadwick
This problem sounds awfully like what I reported in a recent PR. Now I'm starting to have my doubts as to whether or not my hardware was indeed at fault... http://www.freebsd.org/cgi/query-pr.cgi?pr=103435 -- | Jeremy Chadwick jdc at parodius.com | | Parodius

ffs snapshot lockup

2006-09-22 Thread Vivek Khera
Yesterday we upgraded an amd64 system from 6.1 to 6.2-PRE from a cvsup of September 20 evening. The overnight nightly dump to tape went off normally. This afternoon when the sysadmin was performing the level0 backup, the ffs_mksnap program locked up the entire system. It wasn't the

Re: ffs snapshot lockup

2006-09-22 Thread Kris Kennaway
On Fri, Sep 22, 2006 at 02:55:34PM -0400, Vivek Khera wrote: Yesterday we upgraded an amd64 system from 6.1 to 6.2-PRE from a cvsup of September 20 evening. The overnight nightly dump to tape went off normally. This afternoon when the sysadmin was performing the level0 backup, the

Re: ffs snapshot lockup

2006-09-22 Thread Vivek Khera
On Sep 22, 2006, at 3:03 PM, Kris Kennaway wrote: I know this is very un-useful as a bug report, but I'm putting this out in case anyone else has noticed this. It has never happened to us before when we were running 5.4 and then 6.1 on this box. Nothing else changed on the system during this

Re: ffs snapshot lockup

2006-09-22 Thread Kris Kennaway
On Fri, Sep 22, 2006 at 04:24:41PM -0400, Vivek Khera wrote: On Sep 22, 2006, at 3:03 PM, Kris Kennaway wrote: I know this is very un-useful as a bug report, but I'm putting this out in case anyone else has noticed this. It has never happened to us before when we were running 5.4 and then