Hello all,

I am not sure my original mail got through to the list
(I haven't received it back), so I attach it below.

Anyhow, now I have a saved kernel crash dump of the system
panicking when it tries to - I believe - deferred-release
the corrupted deduped blocks which are no longer referenced
by the userdata/blockpointer tree.

As I previously wrote in my thread on unfixeable corruptions
vs. dedup, there may be some single- or multiple-block
references from DDT pointing into "nowhere" - the block
pointed to contains non-matching data, and now the BP
tree does not point to those blocks either. And there
are other blocks with same recorded checksums (repaired
from another source).

I am not really sure what and how breaks the system into
panic. I would like to analyze the kernel dump to see if
my assumptions is true, and perhaps catch the data block
reference from the dump - but I don't know how and ask
for a crash-course ;)

Ultimately I think that if the current system fails to
safely process whatever data it does find on-disk, then
it may be possible to fix that data (or the system) so
that the error is not there and fatal failures don't
occur either.

In the current state of things the pool is practically
unimportable - within a minute after import the system

I did not try rolling back yet... and without deferred
release of blocks (i.e. no guarantee that I can safely
roll back to a consistent BP tree where all nodes exist),
I am actually reluctant to try rollbacks until required to...


2012-02-04 4:28, Jim Klimov wrote:
I got the machine with my 6-disk raidz2 pool booted again,
into oi_151a, but it reboots soon after importing the pool.
Kernel hits a NULL pointer dereference in DDT-related
routines and crashes.

According to fmdump, error and stacktrace is more or less
the same each time. It seems that "repairing" corrupted
deduped data by overwriting blocks or whole files with
good copies did not go too well, even though all of my
deduped datasets now use "dedup=verify" mode.

The uptimes are too short for "savecore" to complete :(
I'll try to catch the system at a good moment to prevent
"pool"'s importing and hope to get the kernel dump.

What should I look for with ZDB, MDB or in dump files?

Any suggestions how to analyze and ultimately fix this
problem on-disk (without destroying and remaking the pool)?

Here's the latest fmdump:

# fmdump -Vp -u 4f6725c1-509f-eba4-8774-e627e1925461
Feb 04 2012 04:11:20.930300000 4f6725c1-509f-eba4-8774-e627e1925461

Feb 04 04:11:11.3891 ireport.os.sunos.panic.dump_pending_on_device

nvlist version: 0
version = 0x0
class = list.suspect
uuid = 4f6725c1-509f-eba4-8774-e627e1925461
code = SUNOS-8000-KL
diag-time = 1328314280 870353
de = fmd:///module/software-diagnosis
fault-list-sz = 0x1
fault-list = (array of embedded nvlists)
(start fault-list[0])
nvlist version: 0
version = 0x0
class = defect.sunos.kernel.panic
certainty = 0x64
asru =
resource =
savecore-succcess = 0
os-instance-uuid = 4f6725c1-509f-eba4-8774-e627e1925461
panicstr = BAD TRAP: type=e (#pf Page fault) rp=ffffff0010a5e920 addr=30
occurred in module "zfs" due to a NULL pointer dereference
panicstack = unix:die+dd () | unix:trap+1799 () | unix:cmntrap+e6 () |
zfs:ddt_phys_decref+c () | zfs:zio_ddt_free+5c () | zfs:zio_execute+8d
() | genunix:taskq_thread+285 () | unix:thread_start+8 () |
crashtime = 1328314064
panic-time = February 4, 2012 04:07:44 AM MSK MSK
(end fault-list[0])

fault-status = 0x1
severity = Major
__ttl = 0x1
__tod = 0x4f2c77a8 0x37734060

And here's some dmesg leading up to that fmdump reference:

Feb 4 04:11:10 bofh-sol zfs: [ID 249136 kern.info] imported version 28
pool pool using 28
Feb 4 04:11:11 bofh-sol savecore: [ID 570001 auth.error] reboot after
panic: BAD TRAP: type=e (#pf Page fault) rp=ffffff0010a5e920 addr=30
occurred in module "zfs" due to a NULL pointer dereference
Feb 4 04:11:11 bofh-sol savecore: [ID 564761 auth.error] Panic crashdump
pending on dump device but dumpadm -n in effect; run savecore(1M)
manually to extract. Image UUID 4f6725c1-509f-eba4-8774-e627e1925461.
Feb 4 04:11:13 bofh-sol unix: [ID 954099 kern.info] NOTICE: IRQ17 is
being shared by drivers with different interrupt levels.
Feb 4 04:11:13 bofh-sol This may result in reduced system performance.
Feb 4 04:11:20 bofh-sol fmd: [ID 377184 daemon.error] SUNW-MSG-ID:
SUNOS-8000-KL, TYPE: Defect, VER: 1, SEVERITY: Major
Feb 4 04:11:20 bofh-sol EVENT-TIME: Sat Feb 4 04:11:20 MSK 2012
Feb 4 04:11:20 bofh-sol PLATFORM: System-Product-Name, CSN:
System-Serial-Number, HOSTNAME: bofh-sol
Feb 4 04:11:20 bofh-sol SOURCE: software-diagnosis, REV: 0.1
Feb 4 04:11:20 bofh-sol EVENT-ID: 4f6725c1-509f-eba4-8774-e627e1925461
Feb 4 04:11:20 bofh-sol DESC: The system has rebooted after a kernel
panic. Refer to http://sun.com/msg/SUNOS-8000-KL for more information.
Feb 4 04:11:20 bofh-sol AUTO-RESPONSE: The failed system image was
dumped to the dump device. If savecore is enabled (see dumpadm(1M)) a
copy of the dump will be written to the savecore directory .
Feb 4 04:11:20 bofh-sol IMPACT: There may be some performance impact
while the panic is copied to the savecore directory. Disk space usage by
panics can be substantial.
Feb 4 04:11:20 bofh-sol REC-ACTION: If savecore is not enabled then
please take steps to preserve the crash image.
Feb 4 04:11:20 bofh-sol Use 'fmdump -Vp -u
4f6725c1-509f-eba4-8774-e627e1925461' to view more panic detail. Please
refer to the knowledge article for additional information.

Thanks in advance,
//Jim Klimov

zfs-discuss mailing list

Reply via email to