softupdates panic?
Hello,
Today -current, 100% reproducable.
Script started on Wed Mar 26 18:17:44 2003
golf# gdb kernel.2603 -k vmcore.29
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for details.
This GDB was configured as i386-undermydesk-freebsd...
panic: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x000a
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc02efaf0
stack pointer = 0x10:0xe1dfea88
frame pointer = 0x10:0xe1dfea8c
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 3463 (zsh)
trap number = 12
panic: page fault
Uptime: 5m52s
Dumping 511 MB
ata0: resetting devices ..
done
[CTRL-C to abort] [CTRL-C to abort] [CTRL-C to abort] 16[CTRL-C to abort] [CTRL-C to
abort] [CTRL-C to abort] 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272
288 304 320 336 352 368 384 400 416 432 448 464 480 496
---
warning: cannot find file for module nvidia.ko
Error while mapping shared library sections:
nvidia.ko: No such file or directory.
Reading symbols from /boot/kernel/if_fxp.ko...done.
Loaded symbols for /boot/kernel/if_fxp.ko
Reading symbols from /boot/kernel/miibus.ko...done.
Loaded symbols for /boot/kernel/miibus.ko
Reading symbols from /boot/kernel/random.ko...done.
Loaded symbols for /boot/kernel/random.ko
Error while reading shared library symbols:
nvidia.ko: No such file or directory.
Reading symbols from /boot/kernel/linux.ko...done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/sysvshm.ko...done.
Loaded symbols for /boot/kernel/sysvshm.ko
Reading symbols from /boot/kernel/sysvsem.ko...done.
Loaded symbols for /boot/kernel/sysvsem.ko
Reading symbols from /boot/kernel/sysvmsg.ko...done.
Loaded symbols for /boot/kernel/sysvmsg.ko
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/ipfw.ko...done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/nfsserver.ko...done.
Loaded symbols for /boot/kernel/nfsserver.ko
Reading symbols from /boot/kernel/blank_saver.ko...done.
Loaded symbols for /boot/kernel/blank_saver.ko
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:239
239 dumping++;
(kgdb) bt full
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:239
No locals.
#1 0xc023614a in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:371
No locals.
#2 0xc02363f3 in panic () at /usr/src/sys/kern/kern_shutdown.c:542
td = (struct thread *) 0xc477b3c0
bootopt = 260
newpanic = 1
buf = page fault, '\0' repeats 245 times
#3 0xc0353bf2 in trap_fatal (frame=0xe1dfea48, eva=0)
at /usr/src/sys/i386/i386/trap.c:843
code = 16
type = 12
ss = 16
esp = 0
softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27,
ssd_dpl = 0, ssd_p = 1, ssd_xx = 3, ssd_xx1 = 2, ssd_def32 = 1, ssd_gran = 1}
#4 0xc03538d2 in trap_pfault (frame=0xe1dfea48, usermode=0, eva=4294901770)
at /usr/src/sys/i386/i386/trap.c:757
va = 4294901760
vm = (struct vmspace *) 0x0
map = (struct vm_map *) 0xc082f000
rv = 1
---Type return to continue, or q return to quit---
ftype = 1 '\001'
td = (struct thread *) 0xc477b3c0
p = (struct proc *) 0xc478d800
#5 0xc035346d in trap (frame=
{tf_fs = 24, tf_es = -1069547504, tf_ds = -505479152, tf_edi = 0, tf_esi = 0,
tf_ebp = -505419124, tf_isp = -505419148, tf_ebx = -65536, tf_edx = -998168100, tf_ecx
= -65536, tf_eax = -998168192, tf_trapno = 12, tf_err = 0, tf_eip = -1070662928, tf_cs
= 8, tf_eflags = 66195, tf_esp = 0, tf_ss = -505419092})
at /usr/src/sys/i386/i386/trap.c:444
td = (struct thread *) 0xc477b3c0
p = (struct proc *) 0xc478d800
sticks = 0
i = 0
ucode = 0
type = 12
code = 0
eva = 4294901770
#6 0xc0344078 in calltrap () at {standard input}:96
No locals.
#7 0xc02f6928 in softdep_update_inodeblock (ip=0x, bp=0xce643030,
waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4681
inodedep = (struct inodedep *) 0xc4812980
wk = (struct worklist *) 0x
---Type return to continue, or q return to quit---
error = 0
gotit = -65536
#8 0xc02e7702 in ffs_update (vp=0xc4636490, waitfor=0)
at /usr/src/sys/ufs/ffs/ffs_inode.c:109
fs = (struct fs *) 0xc41a7000
bp = (struct buf *) 0xce643030
ip = (struct inode *) 0xc4632990
error
Softupdates panic on SMP system
Hi,
I just got a softupdates related panic on my dual PPro during
buildworld. Sources from July, 4th. Build with -j4.
I'll keep the dump if there's any need.
#0 boot (howto=260) at ../../kern/kern_shutdown.c:303
#1 0xc0177a29 in panic (fmt=0xc0298c54 "from debugger")
at ../../kern/kern_shutdown.c:553
#2 0xc0133195 in db_panic (addr=-1071190004, have_addr=0, count=-1,
modif=0xc688e84c "") at ../../ddb/db_command.c:433
#3 0xc0133135 in db_command (last_cmdp=0xc02d4994, cmd_table=0xc02d47f4,
aux_cmd_tablep=0xc0315320) at ../../ddb/db_command.c:333
#4 0xc01331fa in db_command_loop () at ../../ddb/db_command.c:455
#5 0xc01353b7 in db_trap (type=3, code=0) at ../../ddb/db_trap.c:71
#6 0xc026ed51 in kdb_trap (type=3, code=0, regs=0xc688e95c)
at ../../i386/i386/db_interface.c:158
#7 0xc028095c in trap (frame={tf_fs = -964165608, tf_es = -1072103408,
tf_ds = -1039466480, tf_edi = 0, tf_esi = 256, tf_ebp = -964105820,
tf_isp = -964105848, tf_ebx = -1070871776, tf_edx = -1744831553,
tf_ecx = 32, tf_eax = 18, tf_trapno = 3, tf_err = 0,
tf_eip = -1071190004, tf_cs = 8, tf_eflags = 598, tf_esp = -1070836221,
tf_ss = -1070971726}) at ../../i386/i386/trap.c:552
#8 0xc026f00c in Debugger (msg=0xc02a44b2 "panic") at machine/cpufunc.h:64
#9 0xc0177a20 in panic (fmt=0xc02bcb20 "flush_pagedep_deps: flush failed")
at ../../kern/kern_shutdown.c:551
#10 0xc0243521 in flush_pagedep_deps (pvp=0xc656a380, mp=0xc0d05400,
diraddhdp=0xc0f08ba4) at ../../ufs/ffs/ffs_softdep.c:4296
#11 0xc0242de8 in softdep_sync_metadata (ap=0xc688eaf0)
at ../../ufs/ffs/ffs_softdep.c:3980
#12 0xc0246f62 in ffs_fsync (ap=0xc688eaf0) at ../../ufs/ffs/ffs_vnops.c:257
#13 0xc023dce8 in ffs_truncate (vp=0xc656a380, length=10752, flags=4,
cred=0xc0dd6c80, p=0xc6866f60) at vnode_if.h:537
#14 0xc0249d47 in ufs_direnter (dvp=0xc656a380, tvp=0xc66d7980,
dirp=0xc688ec3c, cnp=0xc688ee90, newdirbp=0x0)
at ../../ufs/ufs/ufs_lookup.c:844
#15 0xc024dcf3 in ufs_makeinode (mode=33188, dvp=0xc656a380, vpp=0xc688ee7c,
cnp=0xc688ee90) at ../../ufs/ufs/ufs_vnops.c:2163
#16 0xc024b530 in ufs_create (ap=0xc688ed94) at ../../ufs/ufs/ufs_vnops.c:186
#17 0xc024dd7d in ufs_vnoperate (ap=0xc688ed94)
at ../../ufs/ufs/ufs_vnops.c:2287
#18 0xc01ad4ea in vn_open (ndp=0xc688ee68, flagp=0xc688ee34, cmode=420)
at vnode_if.h:106
#19 0xc01a9685 in open (p=0xc6866f60, uap=0xc688ef80)
at ../../kern/vfs_syscalls.c:992
#20 0xc0281385 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
tf_edi = 8, tf_esi = 672435712, tf_ebp = -1077939112,
tf_isp = -964104236, tf_ebx = 672369188, tf_edx = 672435712,
tf_ecx = 16, tf_eax = 5, tf_trapno = 12, tf_err = 2, tf_eip = 672284676,
tf_cs = 31, tf_eflags = 647, tf_esp = -1077939156, tf_ss = 47})
at ../../i386/i386/trap.c:1126
#21 0xc026f71b in Xint0x80_syscall ()
cannot read proc at 0
(kgdb) up 9
#9 0xc0177a20 in panic (fmt=0xc02bcb20 "flush_pagedep_deps: flush failed")
at ../../kern/kern_shutdown.c:551
551 Debugger ("panic");
(kgdb) list
546 printf("lapic.id = %08x\n", lapic.id);
547 #endif
548
549 #if defined(DDB)
550 if (debugger_on_panic)
551 Debugger ("panic");
552 #endif
553 boot(bootopt);
554 }
555
(kgdb) up
#10 0xc0243521 in flush_pagedep_deps (pvp=0xc656a380, mp=0xc0d05400,
diraddhdp=0xc0f08ba4) at ../../ufs/ffs/ffs_softdep.c:4296
warning: Source file is more recent than executable.
/*
* If we have failed to get rid of all the dependencies
* then something is seriously wrong.
*/
if (dap == LIST_FIRST(diraddhdp))
panic("flush_pagedep_deps: flush failed");
(kgdb) print dap
$1 = (struct diradd *) 0xc0e87200
(kgdb) print *dap
$2 = {da_list = {wk_list = {le_next = 0x0, le_prev = 0xc104e9bc},
wk_type = 10, wk_state = 32777}, da_pdlist = {le_next = 0xc10dada0,
le_prev = 0xc0f08ba4}, da_offset = 8088, da_newinum = 104878, da_un = {
dau_previous = 0xc0f08b80, dau_pagedep = 0xc0f08b80}}
(kgdb) print diraddhdp
$3 = (struct diraddhd *) 0xc0f08ba4
(kgdb) print *diraddhdp
$4 = {lh_first = 0xc0e87200}
(kgdb) print *(diraddhdp-lh_first)
$5 = {da_list = {wk_list = {le_next = 0x0, le_prev = 0xc104e9bc},
wk_type = 10, wk_state = 32777}, da_pdlist = {le_next = 0xc10dada0,
le_prev = 0xc0f08ba4}, da_offset = 8088, da_newinum = 104878, da_un = {
dau_previous = 0xc0f08b80, dau_pagedep = 0xc0f08b80}}
(kgdb) print lk
$7 = {lkt_spl = 0, lkt_held = 60344}
(kgdb) print bp
$8 = (struct buf *) 0xc20b2300
(kgdb) print *bp
$9 = {b_io = {bio_cmd = 2, bio_dev = 0xc0d57a80, bio_blkno = 852656,
bio_offset = 436559872, bio_bcount = 8192,
bio_data = 0xc2b09000 "ยค\201\001", bio_flags = 4, _bio_buf = 0x0,
bio_error = 0, bio_resid = 0, bio_done = 0xc01a09ac bufdonebio,
bio_driver1 = 0x0,
Re: Softupdates panic on SMP system
I did some thinking about this, but no real code inspection, on a walk today- I think what is occurring is that the list of directory updates is getting refreshed from another process while the first process' list is being written out. A quick hack would be to make sure this doesn't happen (no -j4 for loose and gross and incomplete, some kind of worklist ownership token so that when process A wakes back up and looks at the worklist again it can see whether it still owns that list (after all, it drops the locks so it do the write). This is mostly a Kirk issue- he could say much quicker than I about this- I'd file this as a PR - I could fool around with it a little myself, but later this week. -matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: New softupdates panic
:Uniprocessor kernel, checking out /sys from an NFS-mounted repository, :kernel built from sources current as of this afternoon. :(apologies for any typos, no serial console) : :panic: initate_write_inodeblock: already started Checking out from an NFS mounted repository into a local filesystem w/ softupdates enabled? That's correct. ... :panic() :initiate_write_inodeblock() :softdep_disk_io_initiation() :spec_stratety() :bwrite() :vop_stdbwrite() :vop_defaultop() :vfs_bio_awrite() :spec_fsync() :sched_sync() :fork_trampoline() This is definitely new, I've never seen this one before. I'll try to reproduce it on monday when I get back to Berkeley. In the mean time, please see if you can reproduce the panic. If we can get the panic reproducable then I can reproduce it on my test box and handle all the crash dump nicities to hand over to Kirk. I've just built a kernel current as of a few minutes ago, and it's entirely reproducible. What can I do for you now? -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ [EMAIL PROTECTED] \\ and he'll hate you for a lifetime. \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: New softupdates panic
: :panic: initate_write_inodeblock: already started : : Checking out from an NFS mounted repository into a local : filesystem w/ softupdates enabled? : :That's correct. : :I've just built a kernel current as of a few minutes ago, and it's :entirely reproducible. What can I do for you now? Kirk just committed ffs_softdep.c 1.56 a moment ago which he believes will fix this panic. Please update to that version and see if you can reproduce the problem. If you can, the most useful thing you can give me is a debug kernel image and a kernel core. Baring that, a DDB backtrace will do. I have to apologize for us introducing a bunch of new softupdates panics, the fastpath file delete code wasn't supposed to be this complex :-) We only tested the original patch with postmark and didn't try pounding the box with lots of buildworld's before committing the thing. All the bugs found to date have been really easy to fix so I expect softupdates will be stable again by release time. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: New softupdates panic
: :panic: initate_write_inodeblock: already started : : Checking out from an NFS mounted repository into a local : filesystem w/ softupdates enabled? : :That's correct. : :I've just built a kernel current as of a few minutes ago, and it's :entirely reproducible. What can I do for you now? Kirk just committed ffs_softdep.c 1.56 a moment ago which he believes will fix this panic. Please update to that version and see if you can reproduce the problem. If you can, the most useful thing you can give me is a debug kernel image and a kernel core. Baring that, a DDB backtrace will do. I have just done so, and it's now passing the test that was causing it to fail. I can now try to work out what's broken with SMP on this system (seems to be interrupt-allocation related). -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ [EMAIL PROTECTED] \\ and he'll hate you for a lifetime. \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
New softupdates panic
Uniprocessor kernel, checking out /sys from an NFS-mounted repository, kernel built from sources current as of this afternoon. (apologies for any typos, no serial console) panic: initate_write_inodeblock: already started ... panic() initiate_write_inodeblock() softdep_disk_io_initiation() spec_stratety() bwrite() vop_stdbwrite() vop_defaultop() vfs_bio_awrite() spec_fsync() sched_sync() fork_trampoline() -- \\ Give a man a fish, and you feed him for a day. \\ Mike Smith \\ Tell him he should learn how to fish himself, \\ [EMAIL PROTECTED] \\ and he'll hate you for a lifetime. \\ [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Softupdates panic?
The following backtrace comes from a panic I just had on my laptop
that appears to be softupdates-related. Sources are from yesterday,
re-cvsup'ing shows no new fs-related sources updated. At the time I was
in the middle of compiling the Window Maker port (actually compiling the
jpeglib dependency).
ffs_softdep.c version below, others available upon request (along with
kernel.debug/vmcore info):
$FreeBSD: src/sys/contrib/softupdates/ffs_softdep.c,v 1.47 2000/01/11
06:52:35 mckusick Exp $
ata-pci0: Intel PIIX4 ATA-33 controller port 0x3000-0x300f at device 1.1
on pci0
ata-pci0: Busmastering DMA supported
ata0 at 0x01f0 irq 14 on ata-pci0
ad0: FUJITSU MHA2021AT/8210 ATA-3 disk at ata0 as master
ad0: 2067MB (4233600 sectors), 4200 cyls, 16 heads, 63 S/T, 512 B/S
ad0: 16 secs/int, 1 depth queue, WDMA2
Script started on Wed Jan 12 12:25:45 2000
root@pluto:~# gdb -k /sys/compile/PLUTO/kernel.debug /var/crash/vmcore.0
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
IdlePTD 3149824
initial pcb at 28b660
panicstr: from debugger
panic messages:
---
panic: flush_pagedep_deps: flush 3 failed
panic: from debugger
Uptime: 23h19m6s
dumping to dev #ad/0x30001, offset 131072
dump ata0: resetting devices .. done
64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36
35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6
5 4 3 2 1
---
#0 boot (howto=260) at ../../kern/kern_shutdown.c:304
304 dumppcb.pcb_cr3 = rcr3();
(kgdb) bt
#0 boot (howto=260) at ../../kern/kern_shutdown.c:304
#1 0xc01373a5 in panic (fmt=0xc0235bf4 "from debugger")
at ../../kern/kern_shutdown.c:554
#2 0xc011c419 in db_panic (addr=-1071552339, have_addr=0, count=-1,
modif=0xc72a18d0 "") at ../../ddb/db_command.c:433
#3 0xc011c3b9 in db_command (last_cmdp=0xc025e8b8, cmd_table=0xc025e718,
aux_cmd_tablep=0xc02878cc) at ../../ddb/db_command.c:333
#4 0xc011c47e in db_command_loop () at ../../ddb/db_command.c:455
#5 0xc011e507 in db_trap (type=3, code=0) at ../../ddb/db_trap.c:71
#6 0xc0216651 in kdb_trap (type=3, code=0, regs=0xc72a19d8)
at ../../i386/i386/db_interface.c:158
#7 0xc0222088 in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16,
tf_edi = 276254, tf_esi = 256, tf_ebp = -953542112, tf_isp = -953542140,
tf_ebx = -1071353856, tf_edx = 0, tf_ecx = 0, tf_eax = 18,
tf_trapno = 3, tf_err = 0, tf_eip = -1071552339, tf_cs = 8,
tf_eflags = 582, tf_esp = -1071302401, tf_ss = -1071410141})
at ../../i386/i386/trap.c:531
#8 0xc02168ad in Debugger (msg=0xc0239423 "panic") at machine/cpufunc.h:64
#9 0xc013739c in panic (fmt=0xc0247000 "flush_pagedep_deps: flush 3 failed")
at ../../kern/kern_shutdown.c:552
#10 0xc01d01f2 in flush_pagedep_deps (pvp=0xc7391ea0, mp=0xc0ffd800,
diraddhdp=0xc11246a4) at ../../ufs/ffs/ffs_softdep.c:4248
#11 0xc01cfaf8 in softdep_sync_metadata (ap=0xc72a1b6c)
at ../../ufs/ffs/ffs_softdep.c:3934
---Type return to continue, or q return to quit---
#12 0xc01d3a6c in ffs_fsync (ap=0xc72a1b6c) at ../../ufs/ffs/ffs_vnops.c:249
#13 0xc01cab08 in ffs_truncate (vp=0xc7391ea0, length=1024, flags=4,
cred=0xc1006400, p=0xc6d6e100) at vnode_if.h:537
#14 0xc01d6277 in ufs_direnter (dvp=0xc7391ea0, tvp=0xc7378180,
dirp=0xc72a1cb8, cnp=0xc72a1f04, newdirbp=0x0)
at ../../ufs/ufs/ufs_lookup.c:842
#15 0xc01da137 in ufs_makeinode (mode=33188, dvp=0xc7391ea0, vpp=0xc72a1ef0,
cnp=0xc72a1f04) at ../../ufs/ufs/ufs_vnops.c:2159
#16 0xc01d7994 in ufs_create (ap=0xc72a1e10) at ../../ufs/ufs/ufs_vnops.c:183
#17 0xc01da1c1 in ufs_vnoperate (ap=0xc72a1e10)
at ../../ufs/ufs/ufs_vnops.c:2283
#18 0xc0166b1c in vn_open (ndp=0xc72a1edc, fmode=1538, cmode=420)
at vnode_if.h:106
#19 0xc0162de1 in open (p=0xc6d6e100, uap=0xc72a1f80)
at ../../kern/vfs_syscalls.c:994
#20 0xc0222956 in syscall (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
tf_edi = 1, tf_esi = 135000800, tf_ebp = -1077939148,
tf_isp = -953540652, tf_ebx = -1, tf_edx = 134975904,
tf_ecx = -1077939096, tf_eax = 5, tf_trapno = 12, tf_err = 2,
tf_eip = 134692812, tf_cs = 31, tf_eflags = 663, tf_esp = -1077939192,
tf_ss = 47}) at ../../i386/i386/trap.c:1055
#21 0xc0216f56 in Xint0x80_syscall ()
#22 0x8056afd in ?? ()
#23 0x804b9e5 in ?? ()
---Type return to continue, or q return to quit---
#24 0x804a94c in ?? ()
#25 0x804a785 in ?? ()
#26 0x804a8a2 in ?? ()
#27 0x8051ac6 in ?? ()
#28 0x8051a32 in ?? ()
#29 0x80480f9 in ?? ()
(kgdb)
root@pluto:~#
Script done on Wed Jan 12 12:25:56 2000
-
Chris D. Faulhaber - [EMAIL PROTECTED] - [EMAIL PROTECTED]
