-----Original message-----
To:     zfs-discuss@opensolaris.org; 
From:   Deepak Honnalli <deepak.honna...@oracle.com>
Sent:   Wed 28-03-2012 09:12
Subject:        Re: [zfs-discuss] kernel panic during zfs import
> Hi Carsten,
>      This was supposed to be fixed in build 164 of Nevada (6742788). If 
> you are still seeing this
>      issue in S11, I think you should raise a bug with relevant details. 
> As Paul has suggested,
>      this could also be due to incomplete snapshot.
>      I have seen interrupted zfs recv's causing weired bugs.
> Thanks,
> Deepak.

Hi Deepak,

I just spent about an hour (or two) trying to file a bug report regarding the 
issue without success.

Seems to me, that I'm too stupid to use this "MyOracleSupport" portal.

So, as I'm getting paid for keeping systems running and not clicking through 
flash overloaded support portals searching for CSIs, I'm giving the relevant 
information to the list now.

Perhaps, someone at Oracle, reading the list, is able to file a bug report, or 
contact me off list.


Machine A
- Sun X4270 
- Opensolaris Build 111b
- zpool version 14
- primary file server
- sending snapshots via zfs send
- direct attached Sun J4400 SAS JBODs with totally 40 TB storage

Machine B
- Sun X4270
- Solaris 11
- zpool version 33
- mirror server
- receiving snapshots via zfs receive
- FC attached Storagetek FLX280 storage 


After a zfs send/receive run machine B had a hanging zfs receive process. To 
get rid of the process, I rebooted the machine. During reboot the kernel 
panics, resulting in a reboot loop.

To bring up the system, I rebooted single user, removed /etc/zfs/zpool.cache 
and rebooted again.

The damaged pool can imported readonly, giving a warning:

   $>zpool import -o readonly=on san_pool
   cannot set property for 'san_pool/home/someuser': dataset is read-only
   cannot set property for 'san_pool/home/someotheruser': dataset is read-only

The ZFS debugger zdb does not give any additional information:

   $>zdb -d -e san_pool
   Dataset san_pool [ZPL], ID 18, cr_txg 1, 36.0K, 11 objects

The issue can reproduced by trying to import the pool r/w, resulting in a 
kernel panic.

The fmdump utility gives the following information for the relevant UUID:

   $>fmdump -Vp -u 91da1503-74c5-67c2-b7c1-d4e245e4d968
   TIME                           UUID                                 
   Mar 28 2012 12:54:26.563203000 91da1503-74c5-67c2-b7c1-d4e245e4d968 

     TIME                 CLASS                                 ENA
     Mar 28 12:54:24.2698 ireport.os.sunos.panic.dump_available 
     Mar 28 12:54:05.9826 ireport.os.sunos.panic.dump_pending_on_device 

   nvlist version: 0
        version = 0x0
        class = list.suspect
        uuid = 91da1503-74c5-67c2-b7c1-d4e245e4d968
        code = SUNOS-8000-KL
        diag-time = 1332932066 541092
        de = fmd:///module/software-diagnosis
        fault-list-sz = 0x1
        __case_state = 0x1
        topo-uuid = 3b4117e0-0ac7-cde5-b434-b9735176d591
        fault-list = (array of embedded nvlists)
        (start fault-list[0])
        nvlist version: 0
                version = 0x0
                class = defect.sunos.kernel.panic
                certainty = 0x64
                asru = 
                resource = 
                savecore-succcess = 1
                dump-dir = /var/crash
                dump-files = vmdump.0
                os-instance-uuid = 91da1503-74c5-67c2-b7c1-d4e245e4d968
                panicstr = BAD TRAP: type=e (#pf Page fault) 
rp=ffffff002f6dcc50 addr=20 occurred in module "zfs" due to a NULL pointer 
                panicstack = unix:die+d8 () | unix:trap+152b () | 
unix:cmntrap+e6 () | zfs:zap_leaf_lookup_closest+45 () | 
zfs:fzap_cursor_retrieve+cd () | zfs:zap_cursor_retrieve+195 () | 
zfs:zfs_purgedir+4d () |       zfs:zfs_rmnode+57 () | zfs:zfs_zinactive+b4 () | 
zfs:zfs_inactive+1a3 () | genunix:fop_inactive+b1 () | genunix:vn_rele+58 () | 
zfs:zfs_unlinked_drain+a7 () | zfs:zfsvfs_setup+f1 () | zfs:zfs_domount+152 () 
| zfs:zfs_mount+4e3 () | genunix:fsop_mount+22 () | genunix:domount+d2f () | 
genunix:mount+c0 () | genunix:syscall_ap+92 () | unix:brand_sys_sysenter+1cf () 
                crashtime = 1332931339
                panic-time = March 28, 2012 12:42:19 PM CEST CEST
        (end fault-list[0])

        fault-status = 0x1
        severity = Major
        __ttl = 0x1
        __tod = 0x4f72ede2 0x2191cbb8

The 'first view' debugger output looks like:

   mdb unix.0 vmcore.0 
   Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp 
scsi_vhci zfs mpt sd ip hook neti arp usba uhci sockfs qlc fctl s1394 kssl lofs 
random idm sppp crypto sata fcip cpc fcp ufs logindmux ptm ]
   > $c
   zap_leaf_lookup_closest+0x45(ffffff0728eac588, 0, 0, ffffff002f6dcdb0)
   fzap_cursor_retrieve+0xcd(ffffff0728eac588, ffffff002f6dced0, 
   zap_cursor_retrieve+0x195(ffffff002f6dced0, ffffff002f6dcf10)
   zfs_inactive+0x1a3(ffffff0728075080, ffffff0715079548, 0)
   fop_inactive+0xb1(ffffff0728075080, ffffff0715079548, 0)
   zfsvfs_setup+0xf1(ffffff0728c43e00, 1)
   zfs_domount+0x152(ffffff0728cca310, ffffff071de80900)
   zfs_mount+0x4e3(ffffff0728cca310, ffffff0728eab600, ffffff002f6dde20, 
   fsop_mount+0x22(ffffff0728cca310, ffffff0728eab600, ffffff002f6dde20, 
   domount+0xd2f(0, ffffff002f6dde20, ffffff0728eab600, ffffff0715079548, 
   mount+0xc0(ffffff06fcf7fb38, ffffff002f6dde98)

The relevant core files are available for investigation if someone is 
interested in it.


Max Planck Institut fuer marine Mikrobiologie
- Network Administration -
Celsiustr. 1
D-28359 Bremen
Tel.: +49 421 2028568
Fax.: +49 421 2028565
PGP public key:http://www.mpi-bremen.de/Carsten_John.html
mail: cj...@mpi-bremen.de
zfs-discuss mailing list

Reply via email to