Re: RELENG_7/i386: ZFS constant panic on file system writes
On Tue, 7 Apr 2009, Pawel Jakub Dawidek wrote: PJD DM could you please help me a bit with *very* unpleasant situation: one of my PJD DM servers with very large ZFS reboots on most write requests to one (largest, PJD DM which effectively prohibits recreating) ZFS file system with PJD DM PJD DM panic: avl_find() succeeded inside avl_add() PJD PJD Is there a way I can clear the directory in question? Even the latest -current PJD panics when I try to access the directory containing this file. PJD PJD Could you try running 'zpool scrub' on this pool? Nothing better comes PJD to my mind, it looks like some kind of internal inconsistency and PJD hopefully scrub will be able to find it. Could you also show 'zpool status' PJD output? zpool status is showing everything ok: ma...@moose:~ zpool status pool: m state: ONLINE scrub: none requested config: NAMESTATE READ WRITE CKSUM m ONLINE 0 0 0 raidz1ONLINE 0 0 0 ad4hONLINE 0 0 0 ad6hONLINE 0 0 0 ad8hONLINE 0 0 0 ad10h ONLINE 0 0 0 ad12h ONLINE 0 0 0 errors: No known data errors will try scrub, thank you! -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: RELENG_7/i386: ZFS constant panic on file system writes
On Wed, 8 Apr 2009, Dmitry Morozovsky wrote: DM PJD DM could you please help me a bit with *very* unpleasant situation: one of my DM PJD DM servers with very large ZFS reboots on most write requests to one (largest, DM PJD DM which effectively prohibits recreating) ZFS file system with DM PJD DM DM PJD DM panic: avl_find() succeeded inside avl_add() DM PJD DM PJD Is there a way I can clear the directory in question? Even the latest -current DM PJD panics when I try to access the directory containing this file. DM PJD DM PJD Could you try running 'zpool scrub' on this pool? Nothing better comes DM PJD to my mind, it looks like some kind of internal inconsistency and DM PJD hopefully scrub will be able to find it. Could you also show 'zpool status' DM PJD output? DM DM zpool status is showing everything ok: DM DM ma...@moose:~ zpool status DM pool: m DM state: ONLINE DM scrub: none requested DM config: DM DM NAMESTATE READ WRITE CKSUM DM m ONLINE 0 0 0 DM raidz1ONLINE 0 0 0 DM ad4hONLINE 0 0 0 DM ad6hONLINE 0 0 0 DM ad8hONLINE 0 0 0 DM ad10h ONLINE 0 0 0 DM ad12h ONLINE 0 0 0 DM DM errors: No known data errors DM DM will try scrub, thank you! Unfortunately, it does not help: scrub: scrub completed with 0 errors on Wed Apr 8 19:04:51 2009 and then r...@moose:~# ls -la /ar/nfstat/nfc/.bad/200807 total 9089 drwxr-xr-x 3 rscript wheel4 Nov 5 21:01 ./ d- 3 root wheel3 Apr 7 14:29 ../ drwxr-xr-x 2 rscript wheel 36 Apr 2 22:12 daily/ -rw-r--r-- 1 rscript wheel 9207828 Aug 1 2008 total.200807 r...@moose:~# ls -la /ar/nfstat/nfc/.bad/200807/daily/ panic: avl_find() succeeded inside avl_add() cpuid = 2 [-- ma...@localhost detached -- Wed Apr 8 19:28:13 2009] [-- ma...@localhost attached -- Wed Apr 8 19:28:15 2009] [halt sent] KDB: enter: Line break on console [thread pid 153 tid 100152 ] Stopped at kdb_enter_why+0x3a: movl$0,kdb_why db reboot cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 I can set up an account for you to serial console for this server, if it can help... -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: RELENG_7/i386: ZFS constant panic on file system writes
On Fri, 3 Apr 2009, Dmitry Morozovsky wrote: DM Pawel, DM DM could you please help me a bit with *very* unpleasant situation: one of my DM servers with very large ZFS reboots on most write requests to one (largest, DM which effectively prohibits recreating) ZFS file system with DM DM panic: avl_find() succeeded inside avl_add() Is there a way I can clear the directory in question? Even the latest -current panics when I try to access the directory containing this file. DM DM (kgdb) bt DM #0 doadump () at pcpu.h:196 DM #1 0xc0533227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 DM #2 0xc0533535 in panic (fmt=Variable fmt is not available. DM ) at /usr/src/sys/kern/kern_shutdown.c:574 DM #3 0xc0836a20 in avl_add (tree=Variable tree is not available. DM ) at DM /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:635 DM #4 0xc088c39f in zap_lockdir (os=0xc555a590, obj=6108, tx=0x0, lti=RW_READER, DM fatreader=1, zapp=0xfc6907f8) DM at DM /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:231 DM #5 0xc088cc0f in zap_lookup (os=0xc555a590, zapobj=6108, name=0xfc6908bc DM daily.20080701.gz, integer_size=8, num_integers=1, buf=0xfc69083c) DM at DM /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:509 DM #6 0xc089e25d in zfs_dirent_lock (dlpp=0xfc690878, dzp=0xc709f570, DM name=0xfc6908bc daily.20080701.gz, zpp=0xfc690874, flag=Variable flag is DM not available. DM ) DM at DM /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:173 DM #7 0xc089e43e in zfs_dirlook (dzp=0xc709f570, name=0xfc6908bc DM daily.20080701.gz, vpp=0xfc690b5c) DM at DM /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:271 DM #8 0xc08a8653 in zfs_freebsd_lookup (ap=0xfc690a00) DM at DM /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1080 DM #9 0xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a00) at DM vnode_if.c:153 DM #10 0xc05a402c in vfs_cache_lookup (ap=0xfc690a84) at vnode_if.h:83 DM #11 0xc06dc816 in VOP_LOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a84) at DM vnode_if.c:99 DM #12 0xc05aa681 in lookup (ndp=0xfc690b48) at vnode_if.h:57 DM #13 0xc05ab308 in namei (ndp=0xfc690b48) at /usr/src/sys/kern/vfs_lookup.c:215 DM #14 0xc05ba07f in kern_lstat (td=0xc5186af0, path=0xbfbfd088 Address DM 0xbfbfd088 out of bounds, pathseg=UIO_USERSPACE, sbp=0xfc690c18) DM at /usr/src/sys/kern/vfs_syscalls.c:2184 DM #15 0xc05ba22f in lstat (td=0xc5186af0, uap=0xfc690cfc) at DM /usr/src/sys/kern/vfs_syscalls.c:2167 DM #16 0xc06d0288 in syscall (frame=0xfc690d38) at DM /usr/src/sys/i386/i386/trap.c:1090 DM #17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 DM #18 0x0033 in ?? () DM Previous frame inner to this frame (corrupt stack?) DM DM this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from mav@ DM DM Thanks in advance. DM DM -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: RELENG_7/i386: ZFS constant panic on file system writes
On Tue, Apr 07, 2009 at 02:00:28PM +0400, Dmitry Morozovsky wrote: On Fri, 3 Apr 2009, Dmitry Morozovsky wrote: DM Pawel, DM DM could you please help me a bit with *very* unpleasant situation: one of my DM servers with very large ZFS reboots on most write requests to one (largest, DM which effectively prohibits recreating) ZFS file system with DM DM panic: avl_find() succeeded inside avl_add() Is there a way I can clear the directory in question? Even the latest -current panics when I try to access the directory containing this file. Could you try running 'zpool scrub' on this pool? Nothing better comes to my mind, it looks like some kind of internal inconsistency and hopefully scrub will be able to find it. Could you also show 'zpool status' output? -- Pawel Jakub Dawidek http://www.wheel.pl p...@freebsd.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! pgpBHFdAoGWBK.pgp Description: PGP signature
Re: RELENG_7/i386: ZFS constant panic on file system writes
On Thu, Apr 2, 2009 at 22:32, Dmitry Morozovsky ma...@rinet.ru wrote: Pawel, could you please help me a bit with *very* unpleasant situation: one of my servers with very large ZFS reboots on most write requests to one (largest, which effectively prohibits recreating) ZFS file system with panic: avl_find() succeeded inside avl_add() (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc0533227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc0533535 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:574 #3 0xc0836a20 in avl_add (tree=Variable tree is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:635 #4 0xc088c39f in zap_lockdir (os=0xc555a590, obj=6108, tx=0x0, lti=RW_READER, fatreader=1, zapp=0xfc6907f8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:231 #5 0xc088cc0f in zap_lookup (os=0xc555a590, zapobj=6108, name=0xfc6908bc daily.20080701.gz, integer_size=8, num_integers=1, buf=0xfc69083c) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:509 #6 0xc089e25d in zfs_dirent_lock (dlpp=0xfc690878, dzp=0xc709f570, name=0xfc6908bc daily.20080701.gz, zpp=0xfc690874, flag=Variable flag is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:173 #7 0xc089e43e in zfs_dirlook (dzp=0xc709f570, name=0xfc6908bc daily.20080701.gz, vpp=0xfc690b5c) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:271 #8 0xc08a8653 in zfs_freebsd_lookup (ap=0xfc690a00) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1080 #9 0xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a00) at vnode_if.c:153 #10 0xc05a402c in vfs_cache_lookup (ap=0xfc690a84) at vnode_if.h:83 #11 0xc06dc816 in VOP_LOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a84) at vnode_if.c:99 #12 0xc05aa681 in lookup (ndp=0xfc690b48) at vnode_if.h:57 #13 0xc05ab308 in namei (ndp=0xfc690b48) at /usr/src/sys/kern/vfs_lookup.c:215 #14 0xc05ba07f in kern_lstat (td=0xc5186af0, path=0xbfbfd088 Address 0xbfbfd088 out of bounds, pathseg=UIO_USERSPACE, sbp=0xfc690c18) at /usr/src/sys/kern/vfs_syscalls.c:2184 #15 0xc05ba22f in lstat (td=0xc5186af0, uap=0xfc690cfc) at /usr/src/sys/kern/vfs_syscalls.c:2167 #16 0xc06d0288 in syscall (frame=0xfc690d38) at /usr/src/sys/i386/i386/trap.c:1090 #17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #18 0x0033 in ?? () Previous frame inner to this frame (corrupt stack?) this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from mav@ Thanks in advance. Hi Dmitry, I think the line numbers are misleading, see: http://fxr.watson.org/fxr/source/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c?v=FREEBSD7;im=bigexcerpts#L262 Could you build a kernel with -O1 or -O0 perhaps that will help. I have no other clue about your situation except maybe try 8-current? - Marius ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
RELENG_7/i386: ZFS constant panic on file system writes
Pawel, could you please help me a bit with *very* unpleasant situation: one of my servers with very large ZFS reboots on most write requests to one (largest, which effectively prohibits recreating) ZFS file system with panic: avl_find() succeeded inside avl_add() (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc0533227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #2 0xc0533535 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:574 #3 0xc0836a20 in avl_add (tree=Variable tree is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:635 #4 0xc088c39f in zap_lockdir (os=0xc555a590, obj=6108, tx=0x0, lti=RW_READER, fatreader=1, zapp=0xfc6907f8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:231 #5 0xc088cc0f in zap_lookup (os=0xc555a590, zapobj=6108, name=0xfc6908bc daily.20080701.gz, integer_size=8, num_integers=1, buf=0xfc69083c) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:509 #6 0xc089e25d in zfs_dirent_lock (dlpp=0xfc690878, dzp=0xc709f570, name=0xfc6908bc daily.20080701.gz, zpp=0xfc690874, flag=Variable flag is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:173 #7 0xc089e43e in zfs_dirlook (dzp=0xc709f570, name=0xfc6908bc daily.20080701.gz, vpp=0xfc690b5c) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:271 #8 0xc08a8653 in zfs_freebsd_lookup (ap=0xfc690a00) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1080 #9 0xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a00) at vnode_if.c:153 #10 0xc05a402c in vfs_cache_lookup (ap=0xfc690a84) at vnode_if.h:83 #11 0xc06dc816 in VOP_LOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a84) at vnode_if.c:99 #12 0xc05aa681 in lookup (ndp=0xfc690b48) at vnode_if.h:57 #13 0xc05ab308 in namei (ndp=0xfc690b48) at /usr/src/sys/kern/vfs_lookup.c:215 #14 0xc05ba07f in kern_lstat (td=0xc5186af0, path=0xbfbfd088 Address 0xbfbfd088 out of bounds, pathseg=UIO_USERSPACE, sbp=0xfc690c18) at /usr/src/sys/kern/vfs_syscalls.c:2184 #15 0xc05ba22f in lstat (td=0xc5186af0, uap=0xfc690cfc) at /usr/src/sys/kern/vfs_syscalls.c:2167 #16 0xc06d0288 in syscall (frame=0xfc690d38) at /usr/src/sys/i386/i386/trap.c:1090 #17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 #18 0x0033 in ?? () Previous frame inner to this frame (corrupt stack?) this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from mav@ Thanks in advance. -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
