Re: Reliably trigger-able ZFS panic
LI Xin wrote: Hi, The following iozone test case on ZFS would reliably trigger panic: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C It can also be (eventually) triggered by blogbench -c 100 -i 30 -r 50 -w 10 -W 10 and heavy IO load on real multithreaded applications like mysql (both iozone and blogbench are multithreaded). signature.asc Description: OpenPGP digital signature
Re: Reliably trigger-able ZFS panic
On Sun, Mar 02, 2008 at 03:49:03AM -0800, LI Xin wrote: Hi, The following iozone test case on ZFS would reliably trigger panic: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C Thanks, I'll try to reproduce it. [...] #19 0xb55d in z_deflateInit2_ (strm=0xff00042dc8e0, level=70109184, method=68351768, windowBits=68351600, memLevel=76231808, strategy=76231808, version=Cannot access memory at address 0x00040010 ) at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/zmod/deflate.c:318 Can you send me your FS configuration? zfs get all your/file/system I see that you use compression on this dataset? -- Pawel Jakub Dawidek http://www.wheel.pl [EMAIL PROTECTED] http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! pgpBXXAkxJmmK.pgp Description: PGP signature
Re: Reliably trigger-able ZFS panic
Pawel Jakub Dawidek wrote: On Sun, Mar 02, 2008 at 03:49:03AM -0800, LI Xin wrote: Hi, The following iozone test case on ZFS would reliably trigger panic: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C Thanks, I'll try to reproduce it. [...] #19 0xb55d in z_deflateInit2_ (strm=0xff00042dc8e0, level=70109184, method=68351768, windowBits=68351600, memLevel=76231808, strategy=76231808, version=Cannot access memory at address 0x00040010 ) at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/zmod/deflate.c:318 Can you send me your FS configuration? zfs get all your/file/system I see that you use compression on this dataset? It was all default configuration. The pool was a RAID-Z2 without hotspare disk. The box is now running some other tests (not FreeBSD) at our Beijing Lab and we don't have remote hands in the nights, so I'm afraid that I will not be able to provide further information at this moment. Please let me know if the test run will not provoke the problem and I will ask them to see if they can spare the box in the weekend for me. Cheers, -- Xin LI [EMAIL PROTECTED] http://www.delphij.net/ FreeBSD - The Power to Serve! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Reliably trigger-able ZFS panic
Tue, 04 Mar 2008 03:27:35 +0800，Xin LI [EMAIL PROTECTED]: The kernel is FreeBSD fs12.sina.com.cn 7.0-STABLE FreeBSD 7.0-STABLE #0: Sun Mar 2 18:50:05 CST 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/ZFORK amd64 the get all at below: fs12# zfs get all NAME PROPERTY VALUE SOURCE midpool type filesystem - midpool creation Fri Feb 29 15:01 2008 - midpool used 11.1M - midpool available 2.65T - midpool referenced 44.7K - midpool compressratio 1.00x - midpool mountedyes- midpool quota none default midpool reservationnone default midpool recordsize 128K default midpool mountpoint /mnt/ztest local midpool sharenfs offdefault midpool checksum on default midpool compressionoffdefault midpool atime on default midpool deviceson default midpool exec on default midpool setuid on default midpool readonly offdefault midpool jailed offdefault midpool snapdirhidden default midpool aclmodegroupmask default midpool aclinherit secure default midpool canmount on default midpool shareiscsi offdefault midpool xattr offtemporary midpool copies 1 default fs12# zpool get all midpool NAME PROPERTY VALUE SOURCE midpool bootfs- default Pawel Jakub Dawidek wrote: On Sun, Mar 02, 2008 at 03:49:03AM -0800, LI Xin wrote: Hi, The following iozone test case on ZFS would reliably trigger panic: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C Thanks, I'll try to reproduce it. [...] #19 0xb55d in z_deflateInit2_ (strm=0xff00042dc8e0, level=70109184, method=68351768, windowBits=68351600, memLevel=76231808, strategy=76231808, version=Cannot access memory at address 0x00040010 ) at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/zmod/deflate.c:318 Can you send me your FS configuration? zfs get all your/file/system I see that you use compression on this dataset? It was all default configuration. The pool was a RAID-Z2 without hotspare disk. The box is now running some other tests (not FreeBSD) at our Beijing Lab and we don't have remote hands in the nights, so I'm afraid that I will not be able to provide further information at this moment. Please let me know if the test run will not provoke the problem and I will ask them to see if they can spare the box in the weekend for me. Cheers, -- The Power to Serve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Reliably trigger-able ZFS panic
Hi, The following iozone test case on ZFS would reliably trigger panic: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g -i 0 -i 1 -i 2 -i 8 -+p 70 -C Unfortunately the kgdb can not reveal useful backtrace. I have tried KDB_TRACE, but have not yet be able to further investigate it. fs12# kgdb /boot/kernel/kernel.symbols vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol ps_pglobal_lookup] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd. Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 5; apic id = 05 fault virtual address = 0x18 fault code = supervisor read data, page not present instruction pointer = 0x8:0x80763d16 stack pointer = 0x10:0xd94798f0 frame pointer = 0x10:0xd9479920 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 340 (txg_thread_enter) trap number = 12 panic: page fault cpuid = 5 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x17a trap_fatal() at trap_fatal+0x29f trap_pfault() at trap_pfault+0x294 trap() at trap+0x2ea calltrap() at calltrap+0x8 --- trap 0xc, rip = 0x80763d16, rsp = 0xd94798f0, rbp = 0xd9479920 --- dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x26 dmu_objset_sync() at dmu_objset_sync+0x12d dsl_pool_sync() at dsl_pool_sync+0x72 spa_sync() at spa_sync+0x390 txg_sync_thread() at txg_sync_thread+0x12f fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xd9479d30, rbp = 0 --- Uptime: 25m7s Physical memory: 4081 MB Dumping 1139 MB: 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84 68 52 36 20 4 #0 doadump () at pcpu.h:194 194 pcpu.h: No such file or directory. in pcpu.h (kgdb) add-symbol-file /boot/kernel/zfs.ko.symbols add symbol table from file /boot/kernel/zfs.ko.symbols at (y or n) y Reading symbols from /boot/kernel/zfs.ko.symbols...done. (kgdb) where #0 doadump () at pcpu.h:194 #1 0x80277aa8 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 #2 0x80277f07 in panic (fmt=Variable fmt is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0x80465a1f in trap_fatal (frame=0xc, eva=Variable eva is not available. ) at /usr/src/sys/amd64/amd64/trap.c:724 #4 0x80465e04 in trap_pfault (frame=0xd9479840, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641 #5 0x8046677a in trap (frame=0xd9479840) at /usr/src/sys/amd64/amd64/trap.c:410 #6 0x8044babe in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #7 0x80763d16 in ?? () #8 0x0004 in adjust_ace_pair () #9 0x0004 in adjust_ace_pair () #10 0xd94799e0 in ?? () #11 0x80763e7d in ?? () #12 0xff0004275a80 in ?? () #13 0xff00045a1190 in ?? () #14 0x807639b0 in ?? () #15 0x80763f20 in ?? () #16 0xff00042dc800 in ?? () #17 0x0004 in adjust_ace_pair () #18 0xd9479990 in ?? () #19 0xb55d in z_deflateInit2_ (strm=0xff00042dc8e0, level=70109184, method=68351768, windowBits=68351600, memLevel=76231808, strategy=76231808, version=Cannot access memory at address 0x00040010 ) at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/zmod/deflate.c:318 Previous frame inner to this frame (corrupt stack?) -- Xin LI [EMAIL PROTECTED]http://www.delphij.net/ FreeBSD - The Power to Serve! signature.asc Description: OpenPGP digital signature
